00010000:00010000:3.0F:1788572439.894751:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0F:1788572440.927315:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0F:1788572441.946765:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572442.226959:0:17958:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572442.226978:0:17958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572442.226998:0:17958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572442.227017:0:17958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572442.227036:0:17958:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572442.227045:0:17958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572442.227053:0:17958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572442.227058:0:17958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572442.227067:0:17958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1453cb8 00010000:00010000:2.0:1788572442.227071:0:17958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572442.227081:0:17958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200008:2.0:1788572442.228680:0:17958:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572442.228691:0:17958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572442.228710:0:17958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572442.228726:0:17958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572442.228740:0:17958:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572442.228748:0:17958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572442.228761:0:17958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572442.228765:0:17958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572442.228772:0:17958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1453cb8 00010000:00010000:2.0:1788572442.228776:0:17958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572442.228785:0:17958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572442.229031:0:17958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572442.229042:0:17958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572442.229051:0:17958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572442.229061:0:17958:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572442.229067:0:17958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572442.229071:0:17958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572442.229073:0:17958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572442.229077:0:17958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1453bb0 00010000:00010000:2.0:1788572442.229080:0:17958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572442.229086:0:17958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572442.229096:0:17958:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572442.229110:0:17958:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200080:2.0:1788572442.229129:0:17958:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=8080667f arg=7ffe05d1e7f0 00000080:00200008:2.0:1788572442.229144:0:17958:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200000:2.0:1788572442.229909:0:17958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572442.229921:0:17958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572442.229931:0:17958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788572442.230156:0:17958:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572442.230163:0:17958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572442.230169:0:17958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572442.230172:0:17958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572442.230177:0:17958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1453bb0 00010000:00010000:2.0:1788572442.230180:0:17958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788572442.230188:0:17958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788572442.230199:0:17958:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572442.230205:0:17958:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200080:2.0:1788572442.230215:0:17958:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=8080667f arg=7ffe05d1d6d0 00000080:00200008:2.0:1788572442.230224:0:17958:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200000:2.0:1788572442.230565:0:17958:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572442.230577:0:17958:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572442.230587:0:17958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788572442.230604:0:17958:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788572442.230612:0:17958:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572442.230616:0:17958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572442.230619:0:17958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572442.230624:0:17958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1453bb0 00010000:00010000:2.0:1788572442.230627:0:17958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788572442.230634:0:17958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572442.230643:0:17958:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572442.230648:0:17958:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200080:2.0:1788572442.230878:0:17958:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c01466fc arg=7ffe05d1b5b0 00000080:00200008:2.0:1788572442.230978:0:17958:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200080:3.0:1788572442.378684:0:17447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=401866d5 arg=1329980 00800000:00000080:3.0:1788572442.378703:0:17447:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d5 len=24 karg=ffff93a244daaee0 uarg=0000000000000000 00000002:00000080:3.0:1788572442.378712:0:17447:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d5 len=24 karg=ffff93a244daaee0 uarg=0000000000000000 00000100:00100000:3.0:1788572442.378761:0:17447:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:17447:1875453495374592:192.168.202.102@tcp:60:lhsmtool_posix.0 00000100:00100000:3.0:1788572442.378855:0:17447:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572442.386422:0:17447:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901918 gen 1 00000100:00100000:3.0:1788572442.386501:0:17447:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:17447:1875453495374592:192.168.202.102@tcp:60:lhsmtool_posix.0 00000080:00200008:3.0:1788572442.386573:0:17447:0:(file.c:479:ll_file_release()) START file fid:[0x200000002:0x2:0x0](ffff93a25110a1c8), flags 100000 00000002:00100000:3.0:1788572442.386634:0:17447:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249409880 x1875453495359360/t12884901889(12884901889) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572449 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572442.386646:0:17447:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572442.386651:0:17447:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064df80 x1875453495374848/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572442.386673:0:17447:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:17447:1875453495374848:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:3.0:1788572442.386753:0:17447:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572442.402019:0:17447:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:17447:1875453495374848:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:0.0:1788572442.402056:0:17447:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249409880 x1875453495359360/t12884901889(12884901889) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572449 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788572442.402134:0:17447:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200008:1.0F:1788572442.409254:0:17447:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200008:3.0:1788572442.820617:0:17980:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572442.820635:0:17980:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572442.820658:0:17980:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572442.820677:0:17980:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572442.820695:0:17980:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788572442.820703:0:17980:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572442.820720:0:17980:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572442.820724:0:17980:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:3.0:1788572442.820734:0:17980:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1493cb8 00010000:00010000:3.0:1788572442.820739:0:17980:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572442.820747:0:17980:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200008:2.0:1788572442.841658:0:17980:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251395b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572442.841676:0:17980:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251395b88),name=/ 00000002:00010000:2.0:1788572442.841702:0:17980:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572442.841723:0:17980:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572442.841740:0:17980:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572442.841749:0:17980:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572442.841766:0:17980:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) 00000080:00010000:2.0:1788572442.841770:0:17980:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) for lock 0x6d478a807bb114c6 00000080:00010000:2.0:1788572442.841778:0:17980:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114c6 from it ffffbaf6c1493cb8 00010000:00010000:2.0:1788572442.841783:0:17980:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572442.841793:0:17980:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572442.967112:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572443.996891:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572444.502412:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a245044800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:3.0:1788572444.502419:0:8:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff93a244a1e800: disconnect after 21s idle 00000100:00080000:3.0:1788572444.511702:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:3.0:1788572444.511710:0:8:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff93a244b0e800: disconnect after 21s idle 00000100:00080000:3.0:1788572444.511718:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a249169000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:3.0:1788572444.511720:0:8:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff93a244b0e800: disconnect after 21s idle 00000100:00100000:3.0:1788572444.511756:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3f8c1434-73ad-4808-be79-cf461833332c:2355:1875453495374976:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572444.511851:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495375360:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572444.511885:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572444.511891:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495375104:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572444.511915:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2355:1875453495375488:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572444.512173:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495375232:192.168.202.102@tcp:9:kworker.0 00000100:00100000:2.0:1788572444.512222:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc253da9-16dc-43a2-a270-63fc175cf1ea:2352:1875453495375616:192.168.202.102@tcp:9:kworker.0 00000100:00100000:2.0:1788572444.512283:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc253da9-16dc-43a2-a270-63fc175cf1ea:2352:1875453495375744:192.168.202.102@tcp:9:kworker.0 00000100:00100000:3.0:1788572444.516682:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495375104:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572444.516700:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2355:1875453495375488:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572444.516731:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3f8c1434-73ad-4808-be79-cf461833332c:2355:1875453495374976:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572444.523842:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495375360:192.168.202.102@tcp:400:kworker.0 00000100:00080000:2.0:1788572444.529128:0:2352:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff93a25064c700 x1875453495375232/t0(0) o9->lustre-OST0000-osc-ffff93a244a1e800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572454 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572444.529143:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a245044800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572444.529148:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495375232:192.168.202.102@tcp:9:kworker.0 00000100:00080000:2.0:1788572444.541193:0:2352:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff93a25064ca80 x1875453495375616/t0(0) o9->lustre-OST0000-osc-ffff93a244b0e800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572454 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572444.541210:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572444.541214:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc253da9-16dc-43a2-a270-63fc175cf1ea:2352:1875453495375616:192.168.202.102@tcp:9:kworker.0 00000100:00080000:2.0:1788572444.541244:0:2352:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff93a25064df80 x1875453495375744/t0(0) o9->lustre-OST0001-osc-ffff93a244b0e800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572454 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572444.541252:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a249169000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572444.541283:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc253da9-16dc-43a2-a270-63fc175cf1ea:2352:1875453495375744:192.168.202.102@tcp:9:kworker.0 00010000:00010000:3.0:1788572445.026025:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572446.039781:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572447.064724:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572448.090638:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572449.110736:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572449.622686:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495376000:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572449.622792:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2353:1875453495376256:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572449.622827:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788572449.622833:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495375872:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572449.622858:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495376128:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572449.626295:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495375872:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572449.626318:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495376128:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572449.640101:0:2353:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901919 gen 1 00000100:00100000:1.0:1788572449.640121:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495376000:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572449.640135:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2353:1875453495376256:192.168.202.102@tcp:400:kworker.0 00010000:00010000:0.0:1788572450.135725:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572451.160835:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572452.182510:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572453.207150:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572454.232849:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572454.542412:0:18272:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-pcc test 1e: Test RW-PCC with non-root user ==== 21:40:52 (1788572452) 00000100:00100000:3.0:1788572454.760627:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3f8c1434-73ad-4808-be79-cf461833332c:2355:1875453495376384:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.760731:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495376640:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.760762:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572454.760768:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495376512:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.760789:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2355:1875453495376768:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.774545:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495376512:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.774566:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2355:1875453495376768:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.787368:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3f8c1434-73ad-4808-be79-cf461833332c:2355:1875453495376384:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572454.787396:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495376640:192.168.202.102@tcp:400:kworker.0 00000080:00200008:2.0:1788572454.975682:0:18307:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572454.975696:0:18307:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572454.975716:0:18307:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572454.975735:0:18307:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572454.975751:0:18307:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572454.975759:0:18307:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572454.975767:0:18307:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572454.975770:0:18307:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572454.975777:0:18307:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c2027cb8 00010000:00010000:2.0:1788572454.975781:0:18307:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572454.975789:0:18307:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200008:2.0:1788572454.976612:0:18307:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572454.976618:0:18307:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572454.976630:0:18307:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572454.976639:0:18307:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788572454.976658:0:18307:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788572454.976665:0:18307:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572454.976670:0:18307:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572454.976672:0:18307:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572454.976677:0:18307:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c2027cb8 00010000:00010000:2.0:1788572454.976680:0:18307:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788572454.976687:0:18307:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788572454.976771:0:18307:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572454.976778:0:18307:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572454.976785:0:18307:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788572454.976793:0:18307:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788572454.976800:0:18307:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572454.976803:0:18307:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572454.976805:0:18307:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572454.976809:0:18307:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c2027bb0 00010000:00010000:2.0:1788572454.976811:0:18307:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788572454.976825:0:18307:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788572454.976835:0:18307:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572454.976841:0:18307:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200000:1.0:1788572454.979097:0:18308:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572454.979127:0:18308:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572454.979143:0:18308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788572454.979159:0:18308:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788572454.979167:0:18308:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572454.979173:0:18308:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572454.979187:0:18308:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:1.0:1788572454.979195:0:18308:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1d67bb0 00010000:00010000:1.0:1788572454.979199:0:18308:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788572454.979208:0:18308:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572454.979260:0:18308:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572454.979269:0:18308:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200000:1.0:1788572454.979292:0:18308:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572454.979298:0:18308:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572454.979305:0:18308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572454.979318:0:18308:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572454.979324:0:18308:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572454.979329:0:18308:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572454.979331:0:18308:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:1.0:1788572454.979335:0:18308:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1d67bf8 00010000:00010000:1.0:1788572454.979338:0:18308:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572454.979345:0:18308:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572454.979354:0:18308:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572454.979361:0:18308:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.lustre, flags=17 00000080:00200000:1.0:1788572454.979365:0:18308:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x1:0x0](ffff93a251191988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572454.979376:0:18308:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=fid, dir=[0x200000002:0x1:0x0](ffff93a251191988), file ffff93a249452c00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1788572454.979382:0:18308:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fid, dir=[0x200000002:0x1:0x0](ffff93a251191988), intent=open 00000002:00010000:1.0:1788572454.979399:0:18308:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fid,[0x0:0x0:0x0]) in obj [0x200000002:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1788572454.979624:0:18308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18308 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572454.979632:0:18308:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18308 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572454.979644:0:18308:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572454.979659:0:18308:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da2d80 x1875453495377024/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572454.979675:0:18308:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18308 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572454.979699:0:18308:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495377024:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:1.0:1788572454.979865:0:18308:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572454.986563:0:18308:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495377024:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572454.986591:0:18308:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986609:0:18308:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 4/1,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986616:0:18308:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 4/1,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986628:0:18308:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986638:0:18308:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986651:0:18308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 5/2,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986659:0:18308:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 5/2,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986680:0:18308:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572454.986692:0:18308:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a258da2d80 x1875453495377024/t12884901920(12884901920) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572454.986705:0:18308:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da2d80 x1875453495377024/t12884901920(12884901920) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572454.986716:0:18308:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986728:0:18308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 2/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572454.986739:0:18308:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 2/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572454.986748:0:18308:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986755:0:18308:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986762:0:18308:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986781:0:18308:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572454.986792:0:18308:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572454.986803:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572454.986811:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572454.986812:0:18308:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000002:0x2:0x0], updating i_size 16384 00000080:00200000:2.0:1788572454.986818:0:18308:0:(namei.c:147:ll_iget()) got inode: [0x200000002:0x2:0x0](ffff93a25110a1c8): rc = 0 00000080:00010000:2.0:1788572454.986826:0:18308:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000002:0x2:0x0](ffff93a25110a1c8) for lock 0x6d478a807bb114a3 00010000:00010000:3.0:1788572454.986838:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742c6a00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572454.986844:0:16838:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572454.986853:0:16838:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1788572454.986864:0:18308:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000002:0x2:0x0](ffff93a25110a1c8) 00010000:00010000:3.0:1788572454.986864:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 3/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1788572454.986881:0:18308:0:(file.c:1090:ll_file_open()) START file fid:[0x200000002:0x2:0x0](ffff93a25110a1c8), flags 100000 00010000:00010000:3.0:1788572454.986885:0:16838:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572454.986902:0:18308:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da2d80 x1875453495377024/t12884901920(12884901920) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572470 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788572454.986924:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 2/0,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572454.986932:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000080:00010000:2.0:1788572454.986951:0:18308:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114a3 from it ffff93a247a25c60 00010000:00010000:3.0:1788572454.986954:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 1/0,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986957:0:18308:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572454.986972:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6a00/0x6d478a807bb11559 lrc: 0/0,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c0143f expref: -99 pid: 18308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572454.986975:0:18308:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572454.986994:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572454.986997:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572454.987000:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572454.987001:0:18308:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8), inode mode 4040 mask 44 00000100:00100000:3.0:1788572454.987015:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495377152:192.168.202.102@tcp:103:ldlm_bl.0 00000080:00200080:2.0:1788572454.987082:0:18308:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=401866d5 arg=6fb980 00800000:00000080:2.0:1788572454.987095:0:18308:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d5 len=28 karg=ffff93a242be7820 uarg=0000000000000000 00000002:00000080:2.0:1788572454.987106:0:18308:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d5 len=28 karg=ffff93a242be7820 uarg=0000000000000000 00000100:00100000:2.0:1788572454.987130:0:18308:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495377280:192.168.202.102@tcp:59:lhsmtool_posix.0 00000100:00100000:2.0:1788572454.987167:0:18308:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572454.990627:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495377152:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572454.992378:0:18308:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495377280:192.168.202.102@tcp:59:lhsmtool_posix.0 00000080:00200000:1.0:1788572455.095278:0:18316:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572455.095309:0:18316:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572455.095329:0:18316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572455.095348:0:18316:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572455.095358:0:18316:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572455.095366:0:18316:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572455.095369:0:18316:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:1.0:1788572455.095377:0:18316:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1d6fbb0 00010000:00010000:1.0:1788572455.095381:0:18316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572455.095390:0:18316:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572455.097021:0:18316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572455.097039:0:18316:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200080:1.0:1788572455.097160:0:18316:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=8080667f arg=7ffc57c081a0 00000080:00200008:1.0:1788572455.097323:0:18316:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200000:0.0:1788572455.128919:0:18317:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572455.128958:0:18317:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572455.128981:0:18317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788572455.128998:0:18317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:0.0:1788572455.129006:0:18317:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572455.129021:0:18317:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572455.129025:0:18317:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:0.0:1788572455.129034:0:18317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c1b5bbb0 00010000:00010000:0.0:1788572455.129038:0:18317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788572455.129047:0:18317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1788572455.129066:0:18317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:0.0:1788572455.129074:0:18317:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200080:0.0:1788572455.129102:0:18317:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=8080667f arg=7ffc773c3100 00000080:00200008:0.0:1788572455.129118:0:18317:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200000:3.0:1788572455.157050:0:18318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572455.157084:0:18318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572455.157105:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:3.0:1788572455.157122:0:18318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:3.0:1788572455.157131:0:18318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572455.157139:0:18318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572455.157143:0:18318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:3.0:1788572455.157150:0:18318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c0d87b58 00010000:00010000:3.0:1788572455.158501:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:3.0:1788572455.158524:0:18318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:3.0:1788572455.158543:0:18318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572455.158570:0:18318:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1e.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251193a88), flags=13 00000080:00200000:3.0:1788572455.158575:0:18318:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1e.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251193a88), intent=getattr 00000002:00010000:3.0:1788572455.158596:0:18318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572455.158646:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.158653:0:18318:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.158673:0:18318:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572455.158698:0:18318:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18318:1875453495377408:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572455.158786:0:18318:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.162782:0:18318:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18318:1875453495377408:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572455.162809:0:18318:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162824:0:18318:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162832:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162839:0:18318:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162847:0:18318:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162853:0:18318:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be8600 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572455.162858:0:18318:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162865:0:18318:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162873:0:18318:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162895:0:18318:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572455.162903:0:18318:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.162910:0:18318:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11560 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18318 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572455.162931:0:18318:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff93a25064f480 x1875453495377408/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572455.162945:0:18318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572455.162970:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:3.0:1788572455.162984:0:18318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00010000:3.0:1788572455.162995:0:18318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c0d87930 00010000:00010000:3.0:1788572455.162999:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:3.0:1788572455.163007:0:18318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1788572455.163088:0:18318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572455.163103:0:18318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572455.163110:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:3.0:1788572455.163128:0:18318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1788572455.163134:0:18318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572455.163139:0:18318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572455.163142:0:18318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:3.0:1788572455.163147:0:18318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c0d87be8 00010000:00010000:3.0:1788572455.163150:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:3.0:1788572455.164426:0:18318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:3.0:1788572455.164442:0:18318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572455.164451:0:18318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=16392 00000080:00200008:2.0:1788572455.164826:0:18318:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572455.164835:0:18318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.164852:0:18318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.164866:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788572455.164892:0:18318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788572455.164900:0:18318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572455.164905:0:18318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.164908:0:18318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572455.164913:0:18318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c0d87cb8 00010000:00010000:2.0:1788572455.164916:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788572455.164923:0:18318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200008:2.0:1788572455.165622:0:18318:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572455.165627:0:18318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.165634:0:18318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.165643:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788572455.165653:0:18318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788572455.165659:0:18318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572455.165663:0:18318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.165665:0:18318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572455.165669:0:18318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c0d87cb8 00010000:00010000:2.0:1788572455.165678:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788572455.165685:0:18318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788572455.165805:0:18318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.165812:0:18318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.165818:0:18318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788572455.165826:0:18318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788572455.165832:0:18318:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572455.165835:0:18318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.165837:0:18318:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb114bf 00000080:00010000:2.0:1788572455.165840:0:18318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb114bf from it ffffbaf6c0d87bb0 00010000:00010000:2.0:1788572455.165843:0:18318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788572455.165850:0:18318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788572455.165859:0:18318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572455.165866:0:18318:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 300000 00000080:00200080:2.0:1788572455.165882:0:18318:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c03066f0 arg=7fff9709beb0 00000080:00200000:2.0:1788572455.165895:0:18318:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) name=d1e.sanity-pcc stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:2.0:1788572455.165926:0:18318:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788572455.165983:0:18318:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788572455.165990:0:18318:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572455.165993:0:18318:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93400/0x6d478a807bb114bf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01216 expref: -99 pid: 17461 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000020:00100000:2.0:1788572455.166014:0:18318:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572455.166025:0:18318:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940a300 x1875453495377664/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572455.166049:0:18318:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18318:1875453495377664:192.168.202.102@tcp:36:lfs.0 00000100:00100000:2.0:1788572455.166140:0:18318:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572455.170598:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572455.170687:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518430464 00000100:00100000:1.0:1788572455.170698:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518430464 00000100:00100000:1.0:1788572455.170758:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 12 00000100:00100000:1.0:1788572455.170769:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:fc253da9-16dc-43a2-a270-63fc175cf1ea+5:18901:x1875453518430464:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572455.170792:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788572455.172382:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572455.172399:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:fc253da9-16dc-43a2-a270-63fc175cf1ea+5:18901:x1875453518430464:12345-192.168.202.102@tcp:104: Request processed in 1629us (1807us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572455.172422:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 12 00010000:00010000:2.0:1788572455.174498:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572455.174510:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174522:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a24954dc00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572455.174528:0:16837:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174538:0:16837:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174548:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174623:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174629:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572455.174678:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc253da9-16dc-43a2-a270-63fc175cf1ea:2356:1875453495377792:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572455.174778:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174787:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a24954dc00/0x6d478a807bb114c6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01224 expref: -99 pid: 17464 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.174805:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.174806:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.174809:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572455.184863:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:fc253da9-16dc-43a2-a270-63fc175cf1ea:2356:1875453495377792:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572455.185399:0:18318:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18318:1875453495377664:192.168.202.102@tcp:36:lfs.0 00000080:00200000:2.0:1788572455.185456:0:18318:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00200000:2.0:1788572455.185462:0:18318:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00200008:2.0:1788572455.185497:0:18318:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 300000 00000080:00200000:2.0:1788572455.193804:0:18319:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.193831:0:18319:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.193892:0:18319:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 18319 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.193903:0:18319:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18319 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.193913:0:18319:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18319 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.193949:0:18319:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job chmod:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18319:1875453495377920:192.168.202.102@tcp:101:chmod.0 00000100:00100000:2.0:1788572455.194034:0:18319:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.200604:0:18319:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job chmod:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18319:1875453495377920:192.168.202.102@tcp:101:chmod.0 00010000:00010000:2.0:1788572455.200633:0:18319:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200644:0:18319:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200655:0:18319:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200667:0:18319:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200678:0:18319:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200686:0:18319:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200694:0:18319:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.200705:0:18319:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a249408700 x1875453495377920/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572455.200741:0:18319:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200754:0:18319:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572455.200760:0:18319:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572455.200783:0:18319:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:2.0:1788572455.200789:0:18319:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572455.200793:0:18319:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.200796:0:18319:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1c0bb58 00010000:00010000:2.0:1788572455.200799:0:18319:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200807:0:18319:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572455.200831:0:18319:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572455.200852:0:18319:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1e.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251193a88), flags=9 00000080:00200000:2.0:1788572455.200857:0:18319:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1e.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251193a88), intent=getattr 00000002:00010000:2.0:1788572455.200874:0:18319:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572455.200914:0:18319:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18319 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200921:0:18319:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18319 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.200931:0:18319:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18319 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.200961:0:18319:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job chmod:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18319:1875453495378048:192.168.202.102@tcp:101:chmod.0 00000100:00100000:2.0:1788572455.201047:0:18319:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.206871:0:18319:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job chmod:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18319:1875453495378048:192.168.202.102@tcp:101:chmod.0 00010000:00010000:3.0:1788572455.206899:0:18319:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.206931:0:18319:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.206938:0:18319:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.206965:0:18319:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.206977:0:18319:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.206992:0:18319:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.207000:0:18319:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.207008:0:18319:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572455.207028:0:18319:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a249408700 x1875453495378048/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572455.207040:0:18319:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.207050:0:18319:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/4 (0 0) 00000080:00200000:3.0:1788572455.207056:0:18319:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572455.207077:0:18319:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00200000:3.0:1788572455.207081:0:18319:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x4:0x0](ffff93a2511921c8): rc = 0 00000080:00010000:3.0:1788572455.207087:0:18319:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb1156e 00000080:00010000:3.0:1788572455.207100:0:18319:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:3.0:1788572455.207110:0:18319:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1156e from it ffffbaf6c1c0bb20 00010000:00010000:3.0:1788572455.207114:0:18319:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.207125:0:18319:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572455.207175:0:18319:0:(file.c:6296:ll_getattr_dentry()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572455.207182:0:18319:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8),name=d1e.sanity-pcc 00000002:00010000:3.0:1788572455.207201:0:18319:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572455.207210:0:18319:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572455.207221:0:18319:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572455.207671:0:18319:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572455.207679:0:18319:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:3.0:1788572455.207683:0:18319:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb1156e 00000080:00010000:3.0:1788572455.207688:0:18319:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1156e from it ffffbaf6c1c0bcb8 00010000:00010000:3.0:1788572455.207691:0:18319:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572455.207699:0:18319:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572455.207752:0:18319:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572455.207761:0:18319:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572455.207778:0:18319:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572455.207786:0:18319:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572455.207793:0:18319:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572455.207797:0:18319:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572455.207800:0:18319:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572455.207804:0:18319:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1c0bc08 00010000:00010000:3.0:1788572455.207807:0:18319:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572455.207814:0:18319:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572455.207822:0:18319:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572455.207828:0:18319:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=9 00000080:00200008:3.0:1788572455.207843:0:18319:0:(llite_lib.c:2427:ll_setattr_raw()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8) current size 16384, valid attrs 41, mode 41ff, uid 1243267072, gid -27742, new size 0, atime 1788572455.000000000, mtime 1788572455.000000000, ctime 1788572455.000000000 00010000:00010000:3.0:1788572455.207860:0:18319:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572455.207905:0:18319:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572455.207912:0:18319:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572455.207915:0:18319:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217a00/0x6d478a807bb1156e lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01493 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788572455.207952:0:18319:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572455.207961:0:18319:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064f480 x1875453495378304/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572455.207987:0:18319:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job chmod:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18319:1875453495378304:192.168.202.102@tcp:36:chmod.0 00000100:00100000:3.0:1788572455.208091:0:18319:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.218413:0:18319:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job chmod:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18319:1875453495378304:192.168.202.102@tcp:36:chmod.0 00000080:00200000:3.0:1788572455.218449:0:18319:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00200008:3.0:1788572455.218460:0:18319:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8) current size 16384, valid attrs 10000041, mode 41ff, uid 1243267072, gid -27742, new size 0, atime 1788572455.000000000, mtime 1788572455.000000000, ctime 1788572455.000000000, rc 0 00010000:00010000:2.0:1788572455.255356:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572455.276902:0:18325:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.276933:0:18325:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.276953:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572455.276970:0:18325:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572455.276980:0:18325:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572455.276988:0:18325:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.276992:0:18325:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572455.277000:0:18325:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c0d87bf8 00010000:00010000:2.0:1788572455.277004:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572455.277013:0:18325:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572455.277030:0:18325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572455.277044:0:18325:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1e.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251193a88), flags=25 00000080:00200000:2.0:1788572455.277048:0:18325:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1e.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251193a88), intent=0 00000002:00010000:2.0:1788572455.277060:0:18325:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572455.277105:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18325 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.277120:0:18325:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18325 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.277131:0:18325:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18325 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.277156:0:18325:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495378432:192.168.202.102@tcp:101:dd.500 00000100:00100000:2.0:1788572455.277665:0:18325:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.319791:0:18325:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495378432:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572455.319820:0:18325:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.319843:0:18325:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.319851:0:18325:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.319862:0:18325:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.319872:0:18325:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.320102:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.320129:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.320137:0:18325:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.320149:0:18325:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff93a249409180 x1875453495378432/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572455.320162:0:18325:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.320173:0:18325:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/4 (0 0) 00000080:00200000:2.0:1788572455.320179:0:18325:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572455.320209:0:18325:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00200000:2.0:1788572455.320213:0:18325:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x4:0x0](ffff93a2511921c8): rc = 0 00000080:00010000:2.0:1788572455.320218:0:18325:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb11575 00000080:00010000:2.0:1788572455.320275:0:18325:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:2.0:1788572455.320280:0:18325:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11575 from it ffffbaf6c0d87ac8 00010000:00010000:2.0:1788572455.320284:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.320293:0:18325:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572455.320325:0:18325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572455.320348:0:18325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 3 00000080:00200000:2.0:1788572455.320360:0:18325:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), file ffff93a25048a480, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788572455.320369:0:18325:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000002:00010000:2.0:1788572455.320393:0:18325:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open|creat flags 0101102 00010000:00010000:2.0:1788572455.320482:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18325 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.320489:0:18325:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18325 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572455.320499:0:18325:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572455.320504:0:18325:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409180 x1875453495378688/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572455.320513:0:18325:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18325 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.320531:0:18325:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495378688:192.168.202.102@tcp:101:dd.500 00000100:00100000:2.0:1788572455.320654:0:18325:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.328158:0:18325:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495378688:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572455.328198:0:18325:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 4/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328222:0:18325:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328255:0:18325:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328266:0:18325:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328292:0:18325:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328306:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328315:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328323:0:18325:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.328337:0:18325:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff93a249409180 x1875453495378688/t12884901923(12884901923) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00100000:2.0:1788572455.328350:0:18325:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409180 x1875453495378688/t12884901923(12884901923) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572455.328364:0:18325:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572455.328377:0:18325:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328388:0:18325:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572455.328395:0:18325:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2000013a1:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572455.328418:0:18325:0:(namei.c:1628:ll_create_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000080:00200000:2.0:1788572455.328465:0:18325:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00200000:2.0:1788572455.328538:0:18325:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572455.328544:0:18325:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:2.0:1788572455.328547:0:18325:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1157c 00000080:00010000:2.0:1788572455.328556:0:18325:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00200008:2.0:1788572455.328571:0:18325:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 101101 00000002:00100000:2.0:1788572455.328589:0:18325:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409180 x1875453495378688/t12884901923(12884901923) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000080:00010000:2.0:1788572455.328618:0:18325:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1157c from it ffff93a251f0cf00 00010000:00010000:2.0:1788572455.328622:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.328726:0:18325:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572455.328753:0:18325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572455.328853:0:18325:0:(file.c:2650:do_file_write_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788572455.328865:0:18325:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788572455.328871:0:18325:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788572455.328876:0:18325:0:(file.c:2090:ll_file_io_generic()) f1e.sanity-pcc: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788572455.328881:0:18325:0:(file.c:1848:ll_io_set_mirror()) f1e.sanity-pcc: desiginated mirror: 0 00000020:00200000:2.0:1788572455.328889:0:18325:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b7455e21[0x110, 1, [0x2000013a1:0x5:0x0] hash] 00000020:00200000:2.0:1788572455.328890:0:18325:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788572455.329552:0:18325:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572455.329573:0:18325:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788572455.329581:0:18325:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788572455.329585:0:18325:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572455.329602:0:18325:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788572455.329617:0:18325:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572455.329623:0:18325:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010a30, osc lock ffff93a2489c7db0, flags 0x0 00000100:00080000:2.0:1788572455.329714:0:18325:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a244a1e800: connect to NID 192.168.202.102@tcp last attempt 642 00000100:00080000:2.0:1788572455.329728:0:18325:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a244a1e800: import ffff93a245044800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572455.329739:0:18325:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 3407872, lost: 0. 00000100:00100000:2.0:1788572455.329763:0:18325:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495378816/t0(0) o8->lustre-OST0000-osc-ffff93a244a1e800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572455.329851:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495378816:192.168.202.102@tcp:8:dd.500 00010000:00010000:2.0:1788572455.330336:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.330348:0:18325:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.330356:0:18325:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572455.330369:0:18325:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff93a24940a680 x1875453495378944/t0(0) o101->lustre-OST0000-osc-ffff93a244a1e800@192.168.202.102@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000100:00100000:2.0:1788572455.330375:0:18325:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572455.334055:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244a1e800: committing for last_committed 0 gen 2 00000100:00080000:3.0:1788572455.334074:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff93a244a1e800: connect to target with instance 3 00000100:00080000:3.0:1788572455.334083:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff93a244a1e800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572455.334096:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:3.0:1788572455.334099:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a245044800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:3.0:1788572455.334121:0:2352:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572455.334126:0:2352:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:3.0:1788572455.334141:0:2352:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000806bfd80): req@ffff93a24940a680 x1875453495378944/t0(0) o101->lustre-OST0000-osc-ffff93a244a1e800@192.168.202.102@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000100:00100000:3.0:1788572455.334164:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495378816:192.168.202.102@tcp:8:dd.500 00000100:00100000:2.0:1788572455.334794:0:18325:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.337333:0:18325:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495378944:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572455.337358:0:18325:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.337377:0:18325:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.337392:0:18325:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.337432:0:18325:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572455.337445:0:18325:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.337516:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.337524:0:18325:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572455.337551:0:18325:0:(vvp_io.c:1292:vvp_io_write_start()) f1e.sanity-pcc: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788572455.337570:0:18325:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788572455.337616:0:18325:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788572455.337627:0:18325:0:(rw26.c:929:ll_write_end()) page@00000000205e20b9[3 00000000e4eabaca 2 1 00000000f11d9fc7 0 0 0] 00000080:00200000:2.0:1788572455.337631:0:18325:0:(rw26.c:929:ll_write_end()) vmpage @00000000d9921d2c 17ffffc0001001 3:0 ffff93a27425a380 0 lru 00000080:00200000:2.0:1788572455.337751:0:18325:0:(rw26.c:929:ll_write_end()) osc-page@00000000774f9f76 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000ebdcd6a1 00000000461fcaab > 3< 0 0 > 4< 0 0 8 3407872 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572455.337754:0:18325:0:(rw26.c:929:ll_write_end()) end page@00000000205e20b9 00000080:00200000:2.0:1788572455.337755:0:18325:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788572455.337761:0:18325:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788572455.337782:0:18325:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000feb19b5a ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572455.337801:0:18325:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff93a2511954f8, count 1, dirtied 1 00000080:00200000:2.0:1788572455.337805:0:18325:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788572455.337813:0:18325:0:(vvp_io.c:1358:vvp_io_write_start()) f1e.sanity-pcc: write bytes 0, result: 1024 00000020:00010000:2.0:1788572455.337829:0:18325:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba56c8 (ffff93a2516c02d8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572455.337832:0:18325:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba56c8 (ffff93a2516c02d8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572455.337836:0:18325:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010a30 (ffff93a2516c0ab8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572455.337842:0:18325:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.337851:0:18325:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572455.337865:0:18325:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba56c8 (ffff93a2516c02d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572455.337868:0:18325:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010a30 (ffff93a2516c02d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572455.337874:0:18325:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788572455.337881:0:18325:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788572455.337889:0:18325:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572455.337893:0:18325:0:(file.c:2238:ll_file_io_generic()) f1e.sanity-pcc: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788572455.337900:0:18325:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x5:0x0] updating i_size 1024 i_blocks 1 00000080:00200000:2.0:1788572455.337905:0:18325:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788572455.337942:0:18325:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 100001 00000002:00100000:2.0:1788572455.338006:0:18325:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249409180 x1875453495378688/t12884901923(12884901923) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572455.338018:0:18325:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572455.338023:0:18325:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940a680 x1875453495379200/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572455.338044:0:18325:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495379200:192.168.202.102@tcp:35:dd.500 00000100:00100000:2.0:1788572455.338124:0:18325:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.341023:0:18325:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18325:1875453495379200:192.168.202.102@tcp:35:dd.500 00000002:00100000:2.0:1788572455.341071:0:18325:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249409180 x1875453495378688/t12884901923(12884901923) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000080:00200008:2.0:1788572455.455625:0:18331:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572455.455641:0:18331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.455665:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.455684:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572455.455700:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572455.455709:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572455.455717:0:18331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.455720:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572455.455726:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d87cb8 00010000:00010000:2.0:1788572455.455730:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572455.455739:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572455.455765:0:18331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.455771:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.455777:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.455784:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572455.455797:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572455.455801:0:18331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.455804:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572455.455808:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d87b58 00010000:00010000:2.0:1788572455.455811:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.455819:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572455.455828:0:18331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572455.455835:0:18331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=8 00000080:00200008:2.0:1788572455.455841:0:18331:0:(file.c:6296:ll_getattr_dentry()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572455.455845:0:18331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8),name=d1e.sanity-pcc 00000002:00010000:2.0:1788572455.455855:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.455861:0:18331:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 30 res 8589939617/4 (0 0) 00010000:00010000:2.0:1788572455.455902:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.455909:0:18331:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.455918:0:18331:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.455944:0:18331:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495379328:192.168.202.102@tcp:101:lfs.500 00000100:00100000:2.0:1788572455.456037:0:18331:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.461798:0:18331:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495379328:192.168.202.102@tcp:101:lfs.500 00010000:00010000:2.0:1788572455.461829:0:18331:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461847:0:18331:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461855:0:18331:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461867:0:18331:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461878:0:18331:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461891:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461899:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461908:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.461919:0:18331:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a24940bb80 x1875453495379328/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572455.461932:0:18331:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.461960:0:18331:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/4 (0 0) 00000080:00200000:2.0:1788572455.461967:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572455.461991:0:18331:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00010000:2.0:1788572455.461998:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb1158a 00000080:00010000:2.0:1788572455.462005:0:18331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:2.0:1788572455.462009:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1158a from it ffffbaf6c1d87cb8 00010000:00010000:2.0:1788572455.462011:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.462020:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572455.462084:0:18331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.462100:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.462109:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.462120:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572455.462128:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572455.462133:0:18331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.462136:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572455.462140:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d87b58 00010000:00010000:2.0:1788572455.462143:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572455.462160:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572455.462168:0:18331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572455.462175:0:18331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:2.0:1788572455.462179:0:18331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572455.462185:0:18331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=8 00000080:00200008:2.0:1788572455.462192:0:18331:0:(file.c:6296:ll_getattr_dentry()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572455.462196:0:18331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348),name=f1e.sanity-pcc 00000002:00010000:2.0:1788572455.462208:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.462216:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.462224:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572455.464387:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 1 00000080:00010000:2.0:1788572455.464397:0:18331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:2.0:1788572455.464403:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1157c 00000080:00010000:2.0:1788572455.464566:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1157c from it ffffbaf6c1d87cb8 00010000:00010000:2.0:1788572455.464573:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.464583:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572455.467582:0:18331:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788572455.467598:0:18331:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0x5:0x0] 00000008:00010000:3.0:1788572455.467635:0:18331:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572455.467639:0:18331:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a250d9d1b0, osc lock ffff93a258d53560, flags 0x400001002 00010000:00010000:3.0:1788572455.467666:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572455.467684:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788572455.467697:0:18331:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572455.467701:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572455.467710:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:3.0:1788572455.467724:0:18331:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572455.467733:0:18331:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x5:0x0] updating i_size 1024 i_blocks 1 00000020:00010000:3.0:1788572455.467737:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:3.0:1788572455.467739:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.467742:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250d9d1b0 (ffff93a2516196c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.467745:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572455.467747:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250d9d1b0 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572455.467760:0:18331:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572455.467832:0:18331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572455.467849:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572455.467860:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572455.467878:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572455.467886:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572455.467892:0:18331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572455.467896:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572455.467901:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d87bf8 00010000:00010000:3.0:1788572455.467904:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572455.467911:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572455.467923:0:18331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572455.467930:0:18331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:3.0:1788572455.467935:0:18331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:3.0:1788572455.467939:0:18331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=265 00000080:00200000:3.0:1788572455.467943:0:18331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 46 00000080:00200008:3.0:1788572455.467953:0:18331:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 104002 00000002:00010000:3.0:1788572455.467968:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open flags 0100240104003 00010000:00010000:3.0:1788572455.468018:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.468026:0:18331:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572455.468036:0:18331:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572455.468053:0:18331:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064f100 x1875453495379584/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572455.468063:0:18331:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572455.468086:0:18331:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495379584:192.168.202.102@tcp:101:lfs.500 00000100:00100000:3.0:1788572455.469606:0:18331:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.478611:0:18331:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495379584:192.168.202.102@tcp:101:lfs.500 00010000:00010000:3.0:1788572455.478653:0:18331:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478672:0:18331:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478681:0:18331:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478693:0:18331:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478705:0:18331:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478720:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478728:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478753:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572455.478767:0:18331:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064f100 x1875453495379584/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00100000:3.0:1788572455.478780:0:18331:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064f100 x1875453495379584/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:3.0:1788572455.478794:0:18331:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572455.478807:0:18331:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478819:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572455.478830:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572455.478839:0:18331:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478845:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478852:0:18331:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.478881:0:18331:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1788572455.478889:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 0 00000080:00200000:3.0:1788572455.478943:0:18331:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788572455.478949:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1157c 00000080:00010000:3.0:1788572455.478955:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1157c from it ffffbaf6c1d87bf0 00010000:00010000:3.0:1788572455.478958:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572455.478967:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:3.0:1788572455.478990:0:18331:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064f100 x1875453495379584/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00200080:3.0:1788572455.479094:0:18331:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x5:0x0](0000000000000000) cmd=c00c66f3 arg=7ffe845b4f34 00000002:00010000:3.0:1788572455.479114:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x5:0x0], intent: open flags 01004040000002 00010000:00010000:3.0:1788572455.479155:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.479163:0:18331:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572455.479172:0:18331:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572455.479178:0:18331:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064d500 x1875453495379840/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572455.479188:0:18331:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x81000000000000 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572455.479219:0:18331:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495379840:192.168.202.102@tcp:101:lfs.500 00010000:00010000:2.0:1788572455.479637:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572455.479645:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479653:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be8600 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572455.479657:0:16838:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479664:0:16838:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479672:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479691:0:16838:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.479718:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479725:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572455.479744:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479751:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb11591 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01503 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.479777:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.479779:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.479781:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572455.479794:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495379968:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572455.479867:0:18331:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167db40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.484996:0:18331:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495379840:192.168.202.102@tcp:101:lfs.500 00010000:00010000:3.0:1788572455.485025:0:18331:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485038:0:18331:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 4/1,0 mode: --/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485050:0:18331:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 4/1,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485062:0:18331:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 4/1,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485074:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 5/1,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485082:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 5/1,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485090:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 4/0,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572455.485100:0:18331:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff93a25064d500 x1875453495379840/t12884901926(12884901926) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572471 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00100000:3.0:1788572455.485134:0:18331:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064d500 x1875453495379840/t12884901926(12884901926) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572471 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:3.0:1788572455.485149:0:18331:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 3/0,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485163:0:18331:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 2 res 8589939617/5 (0 0) 00000080:00200000:3.0:1788572455.485170:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x5:0x0] suppgids 500 500: rc 0 00000002:00100000:3.0:1788572455.485188:0:18331:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064d500 x1875453495379840/t12884901926(12884901926) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572471 ref 3 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00010000:3.0:1788572455.485196:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb11598 00000080:00010000:3.0:1788572455.485203:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11598 from it ffffbaf6c1d87cc0 00010000:00010000:3.0:1788572455.485206:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 3/0,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572455.485213:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 2/0,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:3.0:1788572455.486801:0:18331:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x5:0x0](0000000000000000) cmd=c00c66f3 arg=1e262a0 00000080:00000400:3.0:1788572455.486875:0:18331:0:(pcc.c:3550:pcc_file_reset_projid()) /tmp/arc1/sanity-pcc.test_1e/: cache fs project quota off, disabling: rc = -95 00000080:00200008:3.0:1788572455.499877:0:18331:0:(file.c:2483:do_file_read_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 0, count: 1048576 00000080:00200000:3.0:1788572455.499913:0:18331:0:(file.c:2090:ll_file_io_generic()) f1e.sanity-pcc: read ppos: 1024, bytes: 1047552 00000080:00200000:3.0:1788572455.499918:0:18331:0:(file.c:1848:ll_io_set_mirror()) f1e.sanity-pcc: desiginated mirror: 0 00000020:00200000:3.0:1788572455.499927:0:18331:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b7455e21[0x110, 2, [0x2000013a1:0x5:0x0] hash] 00000020:00200000:3.0:1788572455.499930:0:18331:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [1024, 1048576) 2048 0 00000080:00200000:3.0:1788572455.499935:0:18331:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572455.499955:0:18331:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 1024 chunk: [1024, 1048576) 1048576, 1047552 00020000:00200000:3.0:1788572455.499969:0:18331:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572455.499988:0:18331:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 255] 00000008:00010000:3.0:1788572455.500004:0:18331:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 1 00000008:00010000:3.0:1788572455.500009:0:18331:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a250d9d1b0, osc lock ffff93a258d53560, flags 0x40000 00010000:00010000:3.0:1788572455.500033:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500048:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500062:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500071:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500079:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:3.0:1788572455.500088:0:18331:0:(vvp_io.c:835:vvp_io_read_start()) f1e.sanity-pcc: read [1024, 1048576) 00000001:00010000:3.0:1788572455.500102:0:18331:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0x5:0x0] 00000008:00010000:3.0:1788572455.500107:0:18331:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572455.500109:0:18331:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a250d9d530, osc lock ffff93a258d52ab0, flags 0x400001002 00010000:00010000:3.0:1788572455.500117:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500126:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:3.0:1788572455.500147:0:18331:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572455.500152:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 8/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500160:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 7/1,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:3.0:1788572455.500167:0:18331:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572455.500174:0:18331:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x5:0x0] updating i_size 1024 i_blocks 1 00000020:00010000:3.0:1788572455.500178:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:3.0:1788572455.500181:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.500184:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250d9d530 (ffff93a2516196c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.500188:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572455.500190:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250d9d530 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572455.500199:0:18331:0:(vvp_io.c:893:vvp_io_read_start()) count:1047552 iocb pos:1024 00000020:00010000:3.0:1788572455.500208:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:3.0:1788572455.500210:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.500213:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250d9d1b0 (ffff93a2516196c8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572455.500217:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572455.500226:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:3.0:1788572455.500276:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572455.500278:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250d9d1b0 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572455.500291:0:18331:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788572455.500301:0:18331:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572455.500305:0:18331:0:(file.c:2238:ll_file_io_generic()) f1e.sanity-pcc: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788572455.500307:0:18331:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1788572455.500356:0:18331:0:(file.c:2483:do_file_read_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 1024, count: 1048576 00000080:00200000:3.0:1788572455.500363:0:18331:0:(file.c:2090:ll_file_io_generic()) f1e.sanity-pcc: read ppos: 1024, bytes: 1048576 00000080:00200000:3.0:1788572455.500365:0:18331:0:(file.c:1848:ll_io_set_mirror()) f1e.sanity-pcc: desiginated mirror: 0 00000020:00200000:3.0:1788572455.500369:0:18331:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b7455e21[0x110, 2, [0x2000013a1:0x5:0x0] hash] 00000020:00200000:3.0:1788572455.500371:0:18331:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [1024, 1049600) 2048 0 00000080:00200000:3.0:1788572455.500374:0:18331:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572455.500381:0:18331:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 1024 chunk: [1024, 1049600) 1049600, 1048576 00020000:00200000:3.0:1788572455.500383:0:18331:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572455.500390:0:18331:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 256] 00000008:00010000:3.0:1788572455.500396:0:18331:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 1 00000008:00010000:3.0:1788572455.500398:0:18331:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a250d9d1b0, osc lock ffff93a258d53560, flags 0x40000 00010000:00010000:3.0:1788572455.503552:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503580:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1052671) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503595:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503605:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503612:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572455.503646:0:18331:0:(vvp_io.c:835:vvp_io_read_start()) f1e.sanity-pcc: read [1024, 1049600) 00000001:00010000:3.0:1788572455.503667:0:18331:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0x5:0x0] 00000008:00010000:3.0:1788572455.503677:0:18331:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572455.503680:0:18331:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a250d9d530, osc lock ffff93a258d52ab0, flags 0x400001002 00010000:00010000:3.0:1788572455.503695:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503704:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:3.0:1788572455.503712:0:18331:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572455.503716:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 8/1,1 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503723:0:18331:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 7/1,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:3.0:1788572455.503728:0:18331:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572455.503735:0:18331:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x5:0x0] updating i_size 1024 i_blocks 1 00000020:00010000:3.0:1788572455.503738:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:3.0:1788572455.503741:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.503743:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250d9d530 (ffff93a2516196c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.503746:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572455.503749:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250d9d530 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572455.503759:0:18331:0:(vvp_io.c:893:vvp_io_read_start()) count:1048576 iocb pos:1024 00000020:00010000:3.0:1788572455.503770:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:3.0:1788572455.503778:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572455.503781:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250d9d1b0 (ffff93a2516196c8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572455.503785:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572455.503794:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00010000:3.0:1788572455.503808:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572455.503810:0:18331:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250d9d1b0 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572455.503817:0:18331:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788572455.503831:0:18331:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572455.503835:0:18331:0:(file.c:2238:ll_file_io_generic()) f1e.sanity-pcc: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788572455.503838:0:18331:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200000:3.0:1788572455.503975:0:18331:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572455.503983:0:18331:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572455.504090:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495380096:192.168.202.102@tcp:1:lfs.500 00000100:00100000:2.0:1788572455.513993:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495379968:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572455.515953:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572455.516354:0:16828:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1788572455.516376:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518430592 00000100:00100000:1.0:1788572455.516385:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518430592 00000100:00100000:1.0:1788572455.516696:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 13 00000100:00100000:1.0:1788572455.516708:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460c7450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:19969:x1875453518430592:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572455.516727:0:16828:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572455.516840:0:16828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572455.516847:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460c7450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:19969:x1875453518430592:12345-192.168.202.102@tcp:104: Request processed in 143us (909us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572455.516855:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 13 00010000:00010000:2.0:1788572455.521437:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572455.521577:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572455.521592:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216400 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:2.0:1788572455.521599:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000008:00010000:2.0:1788572455.521625:0:16837:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000feb19b5a ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000008:00010000:2.0:1788572455.521640:0:16837:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000feb19b5a ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572455.524694:0:16837:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000008:00100000:3.0:1788572455.524791:0:16837:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff93a25064e300 - object 0x240000400:66 offset 0<>1024 00000100:00100000:2.0:1788572455.524875:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495380224:192.168.202.102@tcp:4:lfs.0 00000008:00010000:2.0:1788572455.598626:0:2355:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000feb19b5a ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000008:00010000:2.0:1788572455.598708:0:2355:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000feb19b5a ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800429400000000 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000100:00100000:2.0:1788572455.598738:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495380224:192.168.202.102@tcp:4:lfs.0 00000080:00200000:3.0:1788572455.598856:0:16837:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572455.598923:0:16837:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1788572455.598975:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572455.598983:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572455.599002:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572455.599022:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 000000000c54d643/0x6d478a807bb11583 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c014e7 expref: -99 pid: 18325 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572455.599046:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572455.599049:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572455.599051:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572455.599070:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495380352:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572455.607954:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495380352:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572455.607995:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495380096:192.168.202.102@tcp:1:lfs.500 00010000:00010000:2.0:1788572455.609836:0:18331:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 2/0,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.609862:0:18331:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb11598 lrc: 0/0,0 mode: --/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x977d9e7969c01518 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.609888:0:18331:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064d500 x1875453495379840/t12884901926(12884901926) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572455.609900:0:18331:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572455.609906:0:18331:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940bb80 x1875453495380608/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572455.609926:0:18331:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495380608:192.168.202.102@tcp:35:lfs.500 00000100:00100000:2.0:1788572455.610008:0:18331:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572455.619932:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572455.620031:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518430720 00000100:00100000:0.0:1788572455.620040:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518430720 00000100:00100000:0.0:1788572455.620099:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 14 00000100:00100000:0.0:1788572455.620110:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18903:x1875453518430720:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572455.620129:0:16828:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572455.620206:0:16828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572455.620213:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18903:x1875453518430720:12345-192.168.202.102@tcp:104: Request processed in 106us (294us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572455.620220:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 14 00010000:00010000:2.0:1788572455.620731:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572455.620741:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.620750:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217200 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572455.620755:0:17521:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.620785:0:17521:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.620793:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572455.620818:0:17521:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1788572455.620860:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.620868:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572455.620888:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.620896:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1157c lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c014d2 expref: -99 pid: 18325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572455.620915:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.620917:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572455.620919:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572455.621027:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495380736:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572455.636924:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495380736:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572455.637052:0:18331:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495380608:192.168.202.102@tcp:35:lfs.500 00000002:00100000:2.0:1788572455.637082:0:18331:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064d500 x1875453495379840/t12884901926(12884901926) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00010000:2.0:1788572455.637125:0:18331:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2000013a1:0x5:0x0](ffff93a251195348) 00000002:00010000:2.0:1788572455.637135:0:18331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x5:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572455.637191:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.637217:0:18331:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.637252:0:18331:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18331 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.637286:0:18331:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495380864:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572455.637446:0:18331:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.645704:0:18331:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495380864:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572455.645751:0:18331:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.645770:0:18331:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.645787:0:18331:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a24940bb80 x1875453495380864/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572455.645798:0:18331:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572455.645814:0:18331:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.645828:0:18331:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572455.645836:0:18331:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2000013a1:0x5:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572455.645857:0:18331:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1159f 00000080:00010000:2.0:1788572455.645866:0:18331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1159f from it ffffbaf6c1d87b70 00010000:00010000:2.0:1788572455.645891:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.645901:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.645925:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.645937:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572455.645946:0:18331:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.645980:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.645987:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572455.646001:0:18331:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572455.646015:0:18331:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572455.646048:0:18331:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572455.646060:0:18331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572455.646067:0:18331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572455.646085:0:18331:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572455.646100:0:18331:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572455.646105:0:18331:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572455.646145:0:18331:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572455.646180:0:18331:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:2.0:1788572455.646186:0:18331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572455.646193:0:18331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788572455.647992:0:18331:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 104002 00000002:00100000:2.0:1788572455.648069:0:18331:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064f100 x1875453495379584/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572455.648080:0:18331:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572455.648085:0:18331:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940bb80 x1875453495381120/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572455.648107:0:18331:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495381120:192.168.202.102@tcp:35:lfs.500 00000100:00100000:2.0:1788572455.648204:0:18331:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.656194:0:18331:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18331:1875453495381120:192.168.202.102@tcp:35:lfs.500 00000002:00100000:2.0:1788572455.656250:0:18331:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064f100 x1875453495379584/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00200000:2.0:1788572455.772753:0:18341:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572455.772784:0:18341:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.772804:0:18341:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572455.772821:0:18341:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572455.772830:0:18341:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572455.772838:0:18341:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572455.772842:0:18341:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572455.772849:0:18341:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1dafbf8 00010000:00010000:2.0:1788572455.772853:0:18341:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572455.772862:0:18341:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572455.772877:0:18341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572455.772885:0:18341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=267 00000080:00200000:2.0:1788572455.772892:0:18341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 44 00000080:00200008:2.0:1788572455.772898:0:18341:0:(file.c:1090:ll_file_open()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00010000:2.0:1788572455.772912:0:18341:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:2.0:1788572455.772959:0:18341:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.772968:0:18341:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572455.772987:0:18341:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572455.772999:0:18341:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940bb80 x1875453495381376/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572455.773008:0:18341:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.773029:0:18341:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18341:1875453495381376:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572455.773306:0:18341:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.782983:0:18341:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18341:1875453495381376:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572455.783006:0:18341:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783020:0:18341:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783026:0:18341:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783034:0:18341:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783042:0:18341:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783048:0:18341:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216800 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572455.783053:0:18341:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783075:0:18341:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783083:0:18341:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783089:0:18341:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572455.783096:0:18341:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.783102:0:18341:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115a6 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18341 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.783119:0:18341:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a24940bb80 x1875453495381376/t12884901937(12884901937) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572455.783133:0:18341:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940bb80 x1875453495381376/t12884901937(12884901937) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572455.783144:0:18341:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572455.783162:0:18341:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000002:00100000:2.0:1788572455.783177:0:18341:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940bb80 x1875453495381376/t12884901937(12884901937) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572455.783290:0:18341:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) cmd=901066fc arg=7ffd19358330 00000100:00100000:2.0:1788572455.783361:0:18341:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18341:1875453495381504:192.168.202.102@tcp:34:lfs.0 00000100:00100000:2.0:1788572455.783498:0:18341:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.803907:0:18341:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18341:1875453495381504:192.168.202.102@tcp:34:lfs.0 00000080:00200008:3.0:1788572455.804091:0:18341:0:(file.c:479:ll_file_release()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00100000:3.0:1788572455.804152:0:18341:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940bb80 x1875453495381376/t12884901937(12884901937) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572471 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572455.804164:0:18341:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572455.804168:0:18341:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064ce00 x1875453495381760/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572455.804189:0:18341:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18341:1875453495381760:192.168.202.102@tcp:35:lfs.0 00000100:00100000:3.0:1788572455.804288:0:18341:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167db40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.810403:0:18341:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18341:1875453495381760:192.168.202.102@tcp:35:lfs.0 00000002:00100000:3.0:1788572455.810479:0:18341:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940bb80 x1875453495381376/t12884901937(12884901937) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572455.924148:0:18347:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572455.924176:0:18347:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572455.927799:0:18347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572455.927839:0:18347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572455.927852:0:18347:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:1.0:1788572455.927861:0:18347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572455.927865:0:18347:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:1.0:1788572455.927874:0:18347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1db7bf8 00010000:00010000:1.0:1788572455.927878:0:18347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572455.927887:0:18347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572455.927904:0:18347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572455.927915:0:18347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:1.0:1788572455.927922:0:18347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:1.0:1788572455.927938:0:18347:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), file ffff93a249452480, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1788572455.927944:0:18347:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open 00000002:00010000:1.0:1788572455.927964:0:18347:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open flags 0100001 00010000:00010000:1.0:1788572455.928016:0:18347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18347 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.928023:0:18347:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18347 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572455.928034:0:18347:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572455.928048:0:18347:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da0e00 x1875453495382016/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572455.928063:0:18347:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18347 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572455.928087:0:18347:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18347:1875453495382016:192.168.202.102@tcp:101:dd.500 00000100:00100000:1.0:1788572455.929579:0:18347:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572455.939989:0:18347:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18347:1875453495382016:192.168.202.102@tcp:101:dd.500 00010000:00010000:1.0:1788572455.940018:0:18347:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940034:0:18347:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940041:0:18347:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940053:0:18347:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940064:0:18347:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940077:0:18347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940086:0:18347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572455.940115:0:18347:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572455.940128:0:18347:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a258da0e00 x1875453495382016/t12884901939(12884901939) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00100000:1.0:1788572455.940141:0:18347:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da0e00 x1875453495382016/t12884901939(12884901939) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00010000:1.0:1788572455.940155:0:18347:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572455.940501:0:18347:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572455.940520:0:18347:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:0.0:1788572455.940527:0:18347:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:0.0:1788572455.940676:0:18347:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 1024 00000080:00200000:0.0:1788572455.940681:0:18347:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:0.0:1788572455.940693:0:18347:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 1 -> 1 00000080:00010000:0.0:1788572455.940698:0:18347:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115ad 00000080:00010000:0.0:1788572455.940708:0:18347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00200008:0.0:1788572455.940723:0:18347:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 100000 00000002:00100000:0.0:1788572455.940745:0:18347:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da0e00 x1875453495382016/t12884901939(12884901939) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572471 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000080:00010000:0.0:1788572455.940792:0:18347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115ad from it ffff93a247a25c00 00010000:00010000:0.0:1788572455.940797:0:18347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572455.940806:0:18347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572455.940839:0:18347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 44 00000080:00200008:0.0:1788572455.940871:0:18347:0:(file.c:5214:ll_file_seek()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:0.0:1788572455.940876:0:18347:0:(file.c:5245:ll_file_seek()) COMPLETED file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], offset: 0, type: SEEK_CUR, rc = 0 00000080:00200008:0.0:1788572455.940969:0:18347:0:(file.c:2483:do_file_read_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 0, count: 1024 00000080:00200008:0.0:1788572455.941009:0:18347:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 100000 00000002:00100000:0.0:1788572455.941054:0:18347:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a258da0e00 x1875453495382016/t12884901939(12884901939) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000020:00100000:0.0:1788572455.941064:0:18347:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572455.941076:0:18347:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d3100 x1875453495382272/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572455.941098:0:18347:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18347:1875453495382272:192.168.202.102@tcp:35:dd.500 00000100:00100000:0.0:1788572455.941371:0:18347:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20b40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572455.947617:0:18347:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18347:1875453495382272:192.168.202.102@tcp:35:dd.500 00000002:00100000:3.0:1788572455.947662:0:18347:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a258da0e00 x1875453495382016/t12884901939(12884901939) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572471 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000080:00200000:1.0:1788572456.027447:0:18353:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572456.027543:0:18353:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572456.027564:0:18353:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572456.027582:0:18353:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572456.027590:0:18353:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:1.0:1788572456.027598:0:18353:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572456.027601:0:18353:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:1.0:1788572456.027608:0:18353:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1df3c08 00010000:00010000:1.0:1788572456.027613:0:18353:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572456.027633:0:18353:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572456.027650:0:18353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572456.027660:0:18353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:1.0:1788572456.027665:0:18353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:1.0:1788572456.027669:0:18353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=9 00000080:00200000:1.0:1788572456.027675:0:18353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 2 00000080:00200008:1.0:1788572456.027684:0:18353:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348) current size 1024, valid attrs 8, mode ffff, uid -1591447252, gid -1, new size 256, atime 1788572456.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000 00010000:00010000:1.0:1788572456.027710:0:18353:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572456.027727:0:18353:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572456.027768:0:18353:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572456.027775:0:18353:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572456.027778:0:18353:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7600/0x6d478a807bb115ad lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01573 expref: -99 pid: 18347 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572456.027795:0:18353:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572456.027807:0:18353:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da3800 x1875453495382528/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572456.027832:0:18353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job truncate:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18353:1875453495382528:192.168.202.102@tcp:36:truncate.500 00000100:00100000:1.0:1788572456.028012:0:18353:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572456.031192:0:18353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job truncate:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18353:1875453495382528:192.168.202.102@tcp:36:truncate.500 00000080:00200008:1.0:1788572456.031273:0:18353:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348) current size 1024, valid attrs 10000168, mode ffff, uid -1591447252, gid -1, new size 256, atime 1788572456.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000, rc 0 00000080:00200000:3.0:1788572456.115095:0:18359:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572456.115124:0:18359:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.115142:0:18359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572456.115156:0:18359:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572456.115164:0:18359:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572456.115171:0:18359:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572456.115175:0:18359:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572456.115191:0:18359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1db7c08 00010000:00010000:3.0:1788572456.115196:0:18359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572456.115204:0:18359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572456.115219:0:18359:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572456.116366:0:18359:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:3.0:1788572456.116383:0:18359:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:3.0:1788572456.116427:0:18359:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags=9 00000080:00200000:3.0:1788572456.116433:0:18359:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=getattr 00000002:00010000:3.0:1788572456.116640:0:18359:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572456.116709:0:18359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18359 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.116720:0:18359:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18359 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.116730:0:18359:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18359 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572456.116757:0:18359:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job truncate:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18359:1875453495382656:192.168.202.102@tcp:101:truncate.500 00000100:00100000:3.0:1788572456.116867:0:18359:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.132358:0:18359:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job truncate:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18359:1875453495382656:192.168.202.102@tcp:101:truncate.500 00010000:00010000:3.0:1788572456.132390:0:18359:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132417:0:18359:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132426:0:18359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132439:0:18359:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132451:0:18359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132491:0:18359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132500:0:18359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132507:0:18359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572456.132519:0:18359:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064ce00 x1875453495382656/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'truncate.500' uid:500 gid:500 projid:0 00000002:00010000:3.0:1788572456.132530:0:18359:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572456.132542:0:18359:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132553:0:18359:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:3.0:1788572456.132559:0:18359:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1788572456.132607:0:18359:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 256 00000080:00200000:3.0:1788572456.132611:0:18359:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:3.0:1788572456.132622:0:18359:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 1 -> 1 00000080:00010000:3.0:1788572456.132627:0:18359:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115b4 00000080:00010000:3.0:1788572456.132638:0:18359:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:3.0:1788572456.132641:0:18359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115b4 from it ffffbaf6c1db7bd0 00010000:00010000:3.0:1788572456.132644:0:18359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.132662:0:18359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572456.132705:0:18359:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 2 00000080:00200008:3.0:1788572456.132715:0:18359:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348) current size 256, valid attrs 8, mode ffff, uid -1591447252, gid -1, new size 2048, atime 1788572456.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000 00010000:00010000:3.0:1788572456.132733:0:18359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572456.132746:0:18359:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572456.132784:0:18359:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572456.132789:0:18359:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572456.132790:0:18359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115b4 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c0159d expref: -99 pid: 18359 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572456.132804:0:18359:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572456.132809:0:18359:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064ce00 x1875453495382912/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572456.132824:0:18359:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job truncate:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18359:1875453495382912:192.168.202.102@tcp:36:truncate.500 00000100:00100000:3.0:1788572456.132901:0:18359:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.144004:0:18359:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901942 gen 1 00000100:00100000:3.0:1788572456.144026:0:18359:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a258da2d80 x1875453495377024/t12884901920(12884901920) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572470 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788572456.144084:0:18359:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff93a25064ce00 x1875453495382912/t12884901944(12884901944) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572472 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'truncate.500' uid:500 gid:500 projid:4294967295 00000100:00100000:3.0:1788572456.144098:0:18359:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job truncate:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18359:1875453495382912:192.168.202.102@tcp:36:truncate.500 00000080:00200008:3.0:1788572456.144150:0:18359:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348) current size 256, valid attrs 10000168, mode ffff, uid -1591447252, gid -1, new size 2048, atime 1788572456.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000, rc 0 00000080:00200000:2.0:1788572456.251725:0:18365:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572456.251757:0:18365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572456.251775:0:18365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572456.251790:0:18365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572456.251798:0:18365:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572456.251805:0:18365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572456.251809:0:18365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572456.251816:0:18365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1c0bbf8 00010000:00010000:2.0:1788572456.251820:0:18365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572456.251829:0:18365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572456.251843:0:18365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572456.251851:0:18365:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:2.0:1788572456.251856:0:18365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572456.251868:0:18365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 3 00000080:00200000:2.0:1788572456.251872:0:18365:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), file ffff93a253983c80, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788572456.251881:0:18365:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000002:00010000:2.0:1788572456.251905:0:18365:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x2000013a1:0x7:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open|creat flags 0101102 00010000:00010000:2.0:1788572456.251917:0:18365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.251933:0:18365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.251998:0:18365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.252005:0:18365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572456.252007:0:18365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1158a lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c014f5 expref: -99 pid: 18331 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.252033:0:18365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18365 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.252039:0:18365:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18365 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572456.252051:0:18365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572456.252063:0:18365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408380 x1875453495383168/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572456.252072:0:18365:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18365 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572456.252095:0:18365:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18365:1875453495383168:192.168.202.102@tcp:101:dd.500 00000100:00100000:2.0:1788572456.252377:0:18365:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572456.265135:0:18365:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18365:1875453495383168:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572456.265165:0:18365:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 4/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265184:0:18365:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265213:0:18365:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265259:0:18365:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265273:0:18365:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265289:0:18365:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265297:0:18365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265305:0:18365:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572456.265316:0:18365:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff93a249408380 x1875453495383168/t12884901945(12884901945) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00100000:2.0:1788572456.265329:0:18365:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408380 x1875453495383168/t12884901945(12884901945) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572456.265341:0:18365:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572456.265353:0:18365:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265370:0:18365:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572456.265376:0:18365:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2000013a1:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572456.265431:0:18365:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 2048 00000080:00200000:2.0:1788572456.265436:0:18365:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:2.0:1788572456.265447:0:18365:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 1 -> 1 00000080:00010000:2.0:1788572456.265462:0:18365:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115bb 00000080:00010000:2.0:1788572456.265472:0:18365:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00200008:2.0:1788572456.265486:0:18365:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 101101 00000002:00100000:2.0:1788572456.265508:0:18365:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408380 x1875453495383168/t12884901945(12884901945) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000080:00010000:2.0:1788572456.265538:0:18365:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115bb from it ffff93a251f0c120 00010000:00010000:2.0:1788572456.265543:0:18365:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.265552:0:18365:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572456.265582:0:18365:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 42 00000080:00200008:2.0:1788572456.265594:0:18365:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348) current size 2048, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788572456.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000 00010000:00010000:2.0:1788572456.265615:0:18365:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.265633:0:18365:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572456.265682:0:18365:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.265689:0:18365:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572456.265692:0:18365:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115bb lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c015c7 expref: -99 pid: 18365 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572456.265709:0:18365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572456.265715:0:18365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409c00 x1875453495383424/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572456.265738:0:18365:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18365:1875453495383424:192.168.202.102@tcp:36:dd.500 00000100:00100000:2.0:1788572456.265818:0:18365:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572456.272478:0:18365:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18365:1875453495383424:192.168.202.102@tcp:36:dd.500 00000080:00200008:2.0:1788572456.272566:0:18365:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348) current size 2048, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788572456.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000, rc 0 00000080:00200008:2.0:1788572456.272664:0:18365:0:(file.c:2650:do_file_write_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 0, count: 1024 00000080:00200008:2.0:1788572456.272716:0:18365:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 100001 00000002:00100000:2.0:1788572456.272772:0:18365:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249408380 x1875453495383168/t12884901945(12884901945) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572472 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572456.272784:0:18365:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572456.272789:0:18365:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495383680/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572456.272812:0:18365:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18365:1875453495383680:192.168.202.102@tcp:35:dd.500 00000100:00100000:2.0:1788572456.273154:0:18365:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00010000:00010000:2.0:1788572456.285392:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572456.285632:0:18365:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job dd:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18365:1875453495383680:192.168.202.102@tcp:35:dd.500 00000002:00100000:2.0:1788572456.285662:0:18365:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249408380 x1875453495383168/t12884901945(12884901945) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:0 00000080:00200000:3.0:1788572456.412628:0:18375:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572456.412660:0:18375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.412683:0:18375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788572456.412701:0:18375:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1788572456.412709:0:18375:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572456.412716:0:18375:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572456.412720:0:18375:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572456.412728:0:18375:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e33bf8 00010000:00010000:3.0:1788572456.412732:0:18375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788572456.412742:0:18375:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788572456.412759:0:18375:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572456.412769:0:18375:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=267 00000080:00200000:3.0:1788572456.412782:0:18375:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 44 00000080:00200008:3.0:1788572456.412789:0:18375:0:(file.c:1090:ll_file_open()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00010000:3.0:1788572456.412804:0:18375:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:3.0:1788572456.412863:0:18375:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.412871:0:18375:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572456.412883:0:18375:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572456.412896:0:18375:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064f480 x1875453495383936/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572456.412906:0:18375:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572456.412928:0:18375:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18375:1875453495383936:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572456.413113:0:18375:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.417686:0:18375:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18375:1875453495383936:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572456.417712:0:18375:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417726:0:18375:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417733:0:18375:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417740:0:18375:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417768:0:18375:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417774:0:18375:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be9a00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572456.417778:0:18375:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417785:0:18375:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417792:0:18375:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417798:0:18375:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572456.417805:0:18375:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.417811:0:18375:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c2 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18375 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572456.417831:0:18375:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064f480 x1875453495383936/t12884901949(12884901949) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572456.417843:0:18375:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064f480 x1875453495383936/t12884901949(12884901949) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572456.417856:0:18375:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572456.417878:0:18375:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000002:00100000:3.0:1788572456.417895:0:18375:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064f480 x1875453495383936/t12884901949(12884901949) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1788572456.418028:0:18375:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) cmd=901066fc arg=7fff4c608c70 00000100:00100000:3.0:1788572456.418094:0:18375:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18375:1875453495384064:192.168.202.102@tcp:34:lfs.0 00000100:00100000:3.0:1788572456.418186:0:18375:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.440297:0:18375:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18375:1875453495384064:192.168.202.102@tcp:34:lfs.0 00000080:00200008:3.0:1788572456.440395:0:18375:0:(file.c:479:ll_file_release()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00100000:3.0:1788572456.440468:0:18375:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064f480 x1875453495383936/t12884901949(12884901949) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572456.440480:0:18375:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572456.440485:0:18375:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c380 x1875453495384320/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572456.440505:0:18375:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18375:1875453495384320:192.168.202.102@tcp:35:lfs.0 00000100:00100000:3.0:1788572456.440595:0:18375:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.442864:0:18375:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18375:1875453495384320:192.168.202.102@tcp:35:lfs.0 00000002:00100000:3.0:1788572456.442896:0:18375:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064f480 x1875453495383936/t12884901949(12884901949) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788572456.600685:0:18381:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572456.600699:0:18381:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572456.600719:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.600738:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788572456.600754:0:18381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788572456.600775:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572456.600783:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572456.600786:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572456.600793:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e2bcb8 00010000:00010000:3.0:1788572456.600797:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788572456.600807:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788572456.600835:0:18381:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572456.600841:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.600848:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572456.600857:0:18381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788572456.600863:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572456.600866:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572456.600869:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572456.600872:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e2bb58 00010000:00010000:3.0:1788572456.600875:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572456.600881:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1788572456.600896:0:18381:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572456.600904:0:18381:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=8 00000080:00200008:3.0:1788572456.600911:0:18381:0:(file.c:6296:ll_getattr_dentry()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572456.600915:0:18381:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8),name=d1e.sanity-pcc 00000002:00010000:3.0:1788572456.600926:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.600934:0:18381:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 30 res 8589939617/4 (0 0) 00010000:00010000:3.0:1788572456.600981:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.600988:0:18381:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.600998:0:18381:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572456.601028:0:18381:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18381:1875453495384448:192.168.202.102@tcp:101:lfs.500 00000100:00100000:3.0:1788572456.601139:0:18381:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.614368:0:18381:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18381:1875453495384448:192.168.202.102@tcp:101:lfs.500 00010000:00010000:3.0:1788572456.614398:0:18381:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614423:0:18381:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614433:0:18381:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614468:0:18381:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614480:0:18381:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614494:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614502:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614510:0:18381:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572456.614522:0:18381:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064c380 x1875453495384448/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:3.0:1788572456.614535:0:18381:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614544:0:18381:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/4 (0 0) 00000080:00200000:3.0:1788572456.614551:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1788572456.614572:0:18381:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00010000:3.0:1788572456.614577:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb115c9 00000080:00010000:3.0:1788572456.614584:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:3.0:1788572456.614587:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115c9 from it ffffbaf6c1e2bcb8 00010000:00010000:3.0:1788572456.614597:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614607:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572456.614668:0:18381:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572456.614681:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.614691:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1788572456.614702:0:18381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1788572456.614708:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572456.614712:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572456.614714:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572456.614718:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e2bb58 00010000:00010000:3.0:1788572456.614721:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1788572456.614728:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1788572456.614734:0:18381:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572456.614741:0:18381:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:3.0:1788572456.614745:0:18381:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:3.0:1788572456.614759:0:18381:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags=8 00000080:00200000:3.0:1788572456.614763:0:18381:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=getattr 00000002:00010000:3.0:1788572456.614782:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572456.614822:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614829:0:18381:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.614839:0:18381:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572456.614861:0:18381:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18381:1875453495384576:192.168.202.102@tcp:101:lfs.500 00000100:00100000:3.0:1788572456.614949:0:18381:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.622952:0:18381:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18381:1875453495384576:192.168.202.102@tcp:101:lfs.500 00010000:00010000:3.0:1788572456.622984:0:18381:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623001:0:18381:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623008:0:18381:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623021:0:18381:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623032:0:18381:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623045:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623069:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623077:0:18381:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572456.623089:0:18381:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064c380 x1875453495384576/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:3.0:1788572456.623102:0:18381:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572456.623114:0:18381:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623125:0:18381:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:3.0:1788572456.623132:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1788572456.623172:0:18381:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 0 00000080:00200000:3.0:1788572456.623175:0:18381:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:3.0:1788572456.623187:0:18381:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 1 -> 1 00000080:00010000:3.0:1788572456.623191:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115d0 00000080:00010000:3.0:1788572456.623202:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:3.0:1788572456.623206:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115d0 from it ffffbaf6c1e2bb20 00010000:00010000:3.0:1788572456.623209:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.623218:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572456.625936:0:18381:0:(file.c:6296:ll_getattr_dentry()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572456.625949:0:18381:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348),name=f1e.sanity-pcc 00000002:00010000:2.0:1788572456.625978:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572456.625998:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.626014:0:18381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572456.626022:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 1 00000080:00010000:2.0:1788572456.626029:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:2.0:1788572456.626033:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115d0 00000080:00010000:2.0:1788572456.626040:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115d0 from it ffffbaf6c1e2bcb8 00010000:00010000:2.0:1788572456.626044:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572456.626052:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572456.626148:0:18381:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572456.626157:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572456.626164:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788572456.626190:0:18381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788572456.626197:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572456.626202:0:18381:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572456.626205:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572456.626209:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e2bbf8 00010000:00010000:2.0:1788572456.626212:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788572456.626219:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788572456.626418:0:18381:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572456.626428:0:18381:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:2.0:1788572456.626434:0:18381:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572456.626438:0:18381:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=265 00000080:00200000:2.0:1788572456.626442:0:18381:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 46 00000080:00200008:2.0:1788572456.626451:0:18381:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 154402 00000002:00010000:2.0:1788572456.626529:0:18381:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open flags 0104240114403 00010000:00010000:2.0:1788572456.626583:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.626591:0:18381:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572456.626602:0:18381:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572456.626621:0:18381:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495384832/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572456.626630:0:18381:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18381 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572456.626654:0:18381:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18381:1875453495384832:192.168.202.102@tcp:101:lfs.500 00000100:00100000:2.0:1788572456.628794:0:18381:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2489860c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572456.639213:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572456.640111:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518431232 00000100:00100000:1.0:1788572456.640122:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518431232 00000100:00100000:1.0:1788572456.641538:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 15 00000100:00100000:1.0:1788572456.641561:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518431232:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572456.641584:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572456.642644:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572456.642660:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518431232:12345-192.168.202.102@tcp:104: Request processed in 1107us (3456us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572456.642670:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 15 00010000:00010000:3.0:1788572456.645854:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572456.645871:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572456.645884:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be9200 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572456.645890:0:16837:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572456.645899:0:16837:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572456.645909:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.646140:0:16837:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572456.646203:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.646210:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572456.646259:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.646268:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9200/0x6d478a807bb115d0 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01622 expref: -99 pid: 18381 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.646284:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572456.646286:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572456.646288:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572456.646296:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572456.646314:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495384960:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572456.651498:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495384960:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572456.662428:0:18381:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18381:1875453495384832:192.168.202.102@tcp:101:lfs.500 00010000:00010000:2.0:1788572456.662483:0:18381:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662503:0:18381:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 4/1,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662512:0:18381:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 4/1,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662525:0:18381:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 4/1,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662537:0:18381:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 4/1,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662551:0:18381:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 5/1,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662559:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 5/1,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662567:0:18381:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 4/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572456.662579:0:18381:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff93a249408000 x1875453495384832/t12884901951(12884901951) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00100000:2.0:1788572456.662591:0:18381:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408000 x1875453495384832/t12884901951(12884901951) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572456.662612:0:18381:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 3/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662624:0:18381:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 8 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572456.662631:0:18381:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 0 00000080:00200000:2.0:1788572456.662685:0:18381:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 0 00000080:00010000:2.0:1788572456.662690:0:18381:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115d7 00000080:00010000:2.0:1788572456.662698:0:18381:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115d7 from it ffffbaf6c1e2bbf0 00010000:00010000:2.0:1788572456.662701:0:18381:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 3/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572456.662710:0:18381:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572456.662734:0:18381:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408000 x1875453495384832/t12884901951(12884901951) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00200080:2.0:1788572456.662832:0:18381:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x5:0x0](0000000000000000) cmd=c00466fc arg=7ffcd4ca742c 00000080:00200008:2.0:1788572456.662858:0:18381:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 154002 00010000:00010000:2.0:1788572456.662877:0:18381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572456.792466:0:18391:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:3.0:1788572456.792500:0:18391:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572456.792519:0:18391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1788572456.792537:0:18391:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:3.0:1788572456.792546:0:18391:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572456.792554:0:18391:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:3.0:1788572456.792557:0:18391:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:3.0:1788572456.792565:0:18391:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1da7bf8 00010000:00010000:3.0:1788572456.792568:0:18391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1788572456.792584:0:18391:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:3.0:1788572456.792603:0:18391:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572456.792611:0:18391:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=267 00000080:00200000:3.0:1788572456.792617:0:18391:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 44 00000080:00200008:3.0:1788572456.792623:0:18391:0:(file.c:1090:ll_file_open()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00010000:3.0:1788572456.792640:0:18391:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:3.0:1788572456.792699:0:18391:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.792707:0:18391:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572456.792721:0:18391:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572456.792734:0:18391:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c380 x1875453495385216/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572456.792744:0:18391:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572456.792767:0:18391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18391:1875453495385216:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572456.795010:0:18391:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.801642:0:18391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18391:1875453495385216:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572456.801669:0:18391:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801684:0:18391:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801706:0:18391:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801712:0:18391:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801730:0:18391:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801737:0:18391:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be8600 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572456.801742:0:18391:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801749:0:18391:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801756:0:18391:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801771:0:18391:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572456.801777:0:18391:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572456.801784:0:18391:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8600/0x6d478a807bb115de lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18391 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572456.801804:0:18391:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064c380 x1875453495385216/t12884901952(12884901952) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572456.801817:0:18391:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064c380 x1875453495385216/t12884901952(12884901952) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572456.801838:0:18391:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572456.801857:0:18391:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000002:00100000:3.0:1788572456.801872:0:18391:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064c380 x1875453495385216/t12884901952(12884901952) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1788572456.801967:0:18391:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) cmd=901066fc arg=7fffa44eac00 00000100:00100000:3.0:1788572456.802027:0:18391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18391:1875453495385344:192.168.202.102@tcp:34:lfs.0 00000100:00100000:3.0:1788572456.802116:0:18391:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.806049:0:18391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18391:1875453495385344:192.168.202.102@tcp:34:lfs.0 00000080:00200008:3.0:1788572456.806126:0:18391:0:(file.c:479:ll_file_release()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00100000:3.0:1788572456.806188:0:18391:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064c380 x1875453495385216/t12884901952(12884901952) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572456.806201:0:18391:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572456.806207:0:18391:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064f480 x1875453495385600/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572456.806262:0:18391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18391:1875453495385600:192.168.202.102@tcp:35:lfs.0 00000100:00100000:3.0:1788572456.806565:0:18391:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572456.821013:0:18391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18391:1875453495385600:192.168.202.102@tcp:35:lfs.0 00000002:00100000:3.0:1788572456.821048:0:18391:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064c380 x1875453495385216/t12884901952(12884901952) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572456.871104:0:18394:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572456.871136:0:18394:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572456.871155:0:18394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788572456.871171:0:18394:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788572456.871179:0:18394:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572456.871186:0:18394:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572456.871189:0:18394:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572456.871197:0:18394:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e4bb98 00010000:00010000:0.0:1788572456.871200:0:18394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788572456.871210:0:18394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:0.0:1788572456.871225:0:18394:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572456.871269:0:18394:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:0.0:1788572456.871275:0:18394:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:0.0:1788572456.871286:0:18394:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags=8 00000080:00200000:0.0:1788572456.871295:0:18394:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=getattr 00000002:00010000:0.0:1788572456.871307:0:18394:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572456.871357:0:18394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18394 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572456.871365:0:18394:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18394 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572456.871375:0:18394:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18394 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572456.871402:0:18394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18394:1875453495385728:192.168.202.102@tcp:101:lfs.0 00000100:00100000:0.0:1788572456.871518:0:18394:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572456.876569:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572456.876737:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518431360 00000100:00100000:1.0:1788572456.876749:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518431360 00000100:00100000:1.0:1788572456.876909:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 16 00000100:00100000:1.0:1788572456.876919:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518431360:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572456.876938:0:16828:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572456.877009:0:16828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572456.877016:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518431360:12345-192.168.202.102@tcp:104: Request processed in 99us (458us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572456.877023:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 16 00010000:00010000:2.0:1788572456.877081:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572456.877088:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.877096:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216800 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572456.877107:0:17521:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.877115:0:17521:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.877123:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 3/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1788572456.877174:0:17521:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249408000 x1875453495384832/t12884901951(12884901951) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572472 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572456.877186:0:17521:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572456.877191:0:17521:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408380 x1875453495385984/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572456.877211:0:17521:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:2e29ee59-1edd-4938-bd41-1bf1d77c561a:17521:1875453495385984:192.168.202.102@tcp:35:ldlm_bl.0 00000100:00100000:2.0:1788572456.877296:0:17521:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572456.879302:0:17521:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:2e29ee59-1edd-4938-bd41-1bf1d77c561a:17521:1875453495385984:192.168.202.102@tcp:35:ldlm_bl.0 00000002:00100000:2.0:1788572456.879430:0:17521:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249408000 x1875453495384832/t12884901951(12884901951) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00010000:00010000:2.0:1788572456.879563:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 2/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.879573:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572456.879635:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495386112:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572456.881311:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 1/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.881362:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115d7 lrc: 0/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01630 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572456.881384:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572456.881386:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572456.881388:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572456.883919:0:18394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18394:1875453495385728:192.168.202.102@tcp:101:lfs.0 00010000:00010000:1.0:1788572456.883949:0:18394:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.883966:0:18394:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.883974:0:18394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.883985:0:18394:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.883995:0:18394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.884006:0:18394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.884014:0:18394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.884021:0:18394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572456.884032:0:18394:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a2460d2300 x1875453495385728/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572472 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572456.884043:0:18394:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572456.884074:0:18394:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.884084:0:18394:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:1.0:1788572456.884091:0:18394:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572456.884128:0:18394:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 0 00000080:00200000:1.0:1788572456.884132:0:18394:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:1.0:1788572456.884144:0:18394:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 1 -> 1 00000080:00010000:1.0:1788572456.884149:0:18394:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115e5 00000080:00010000:1.0:1788572456.884161:0:18394:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:1.0:1788572456.884164:0:18394:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115e5 from it ffffbaf6c1e4bb60 00010000:00010000:1.0:1788572456.884167:0:18394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572456.884177:0:18394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788572456.888941:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495386112:192.168.202.102@tcp:103:ldlm_bl.0 00000080:00200008:2.0:1788572457.301113:0:18435:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572457.301131:0:18435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572457.301156:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572457.301175:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572457.301191:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572457.301199:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572457.301208:0:18435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572457.301212:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572457.301221:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e83cb8 00010000:00010000:2.0:1788572457.301225:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572457.301263:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572457.301290:0:18435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572457.301296:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572457.301311:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788572457.301320:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572457.301327:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572457.301332:0:18435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572457.301335:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572457.301339:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e83b58 00010000:00010000:2.0:1788572457.301342:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788572457.301349:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788572457.301358:0:18435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572457.301365:0:18435:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=8 00000080:00200008:2.0:1788572457.301372:0:18435:0:(file.c:6296:ll_getattr_dentry()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572457.301376:0:18435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8),name=d1e.sanity-pcc 00000002:00010000:2.0:1788572457.301388:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572457.301396:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572457.301404:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572457.301411:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572457.301433:0:18435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:2.0:1788572457.301436:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb115c9 00000080:00010000:2.0:1788572457.301441:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115c9 from it ffffbaf6c1e83cb8 00010000:00010000:2.0:1788572457.301444:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572457.301453:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572457.301469:0:18435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572457.301476:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572457.301482:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572457.301489:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788572457.301495:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572457.301499:0:18435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572457.301501:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572457.301504:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e83b58 00010000:00010000:2.0:1788572457.301507:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572457.301515:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788572457.301523:0:18435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572457.301533:0:18435:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:2.0:1788572457.301536:0:18435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572457.301541:0:18435:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=8 00000080:00200008:2.0:1788572457.304399:0:18435:0:(file.c:6296:ll_getattr_dentry()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), request_mask 2047, flags 0, foreign 0 00010000:00010000:2.0:1788572457.304610:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788572457.306901:0:18435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348),name=f1e.sanity-pcc 00000002:00010000:0.0:1788572457.306939:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572457.306959:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.306976:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572457.306985:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 1 00000080:00010000:0.0:1788572457.306993:0:18435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:0.0:1788572457.306997:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115e5 00000080:00010000:0.0:1788572457.307004:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115e5 from it ffffbaf6c1e83cb8 00010000:00010000:0.0:1788572457.307008:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572457.307035:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572457.307124:0:18435:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572457.307133:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572457.307142:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1788572457.307152:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1788572457.307158:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:0.0:1788572457.307162:0:18435:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572457.307165:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572457.307169:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e83bf8 00010000:00010000:0.0:1788572457.307172:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1788572457.307179:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1788572457.307187:0:18435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572457.307194:0:18435:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:0.0:1788572457.307198:0:18435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:0.0:1788572457.307202:0:18435:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=265 00000080:00200000:0.0:1788572457.307206:0:18435:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 46 00000080:00200008:0.0:1788572457.307215:0:18435:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 104002 00000002:00010000:0.0:1788572457.307252:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open flags 0104240104003 00010000:00010000:0.0:1788572457.307305:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572457.307313:0:18435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572457.307324:0:18435:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572457.307349:0:18435:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d2300 x1875453495386368/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572457.307359:0:18435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572457.307380:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495386368:192.168.202.102@tcp:101:lfs.500 00000100:00100000:0.0:1788572457.307684:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572457.316643:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572457.316689:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518432384 00000100:00100000:0.0:1788572457.316698:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518432384 00000100:00100000:0.0:1788572457.316750:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 17 00000100:00100000:0.0:1788572457.316759:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518432384:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572457.316777:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572457.316860:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572457.316867:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518432384:12345-192.168.202.102@tcp:104: Request processed in 109us (235us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572457.316875:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 17 00010000:00010000:3.0:1788572457.316937:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572457.316946:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.316954:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742ad800 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572457.316959:0:16837:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.316968:0:16837:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.316976:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.317003:0:16837:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572457.317061:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.317068:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572457.317089:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.317096:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad800/0x6d478a807bb115e5 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01676 expref: -99 pid: 18394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572457.317112:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572457.317128:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572457.317130:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572457.317148:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495386496:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572457.320589:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495386368:192.168.202.102@tcp:101:lfs.500 00010000:00010000:2.0:1788572457.320616:0:18435:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320633:0:18435:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 4/1,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320640:0:18435:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 4/1,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320651:0:18435:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 4/1,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320662:0:18435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 4/1,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320676:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 5/1,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320685:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 5/1,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320692:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 4/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572457.320717:0:18435:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff93a2460d2300 x1875453495386368/t12884901955(12884901955) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00100000:2.0:1788572457.320731:0:18435:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a2460d2300 x1875453495386368/t12884901955(12884901955) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572457.320746:0:18435:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 3/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320759:0:18435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 8 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572457.320767:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 0 00000080:00200000:2.0:1788572457.320806:0:18435:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 0 00000080:00010000:2.0:1788572457.320813:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115ec 00000080:00010000:2.0:1788572457.320820:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115ec from it ffffbaf6c1e83bf0 00010000:00010000:2.0:1788572457.320823:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 3/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.320831:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572457.320855:0:18435:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a2460d2300 x1875453495386368/t12884901955(12884901955) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00200080:2.0:1788572457.320957:0:18435:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x5:0x0](0000000000000000) cmd=c00c66f3 arg=7ffd83d70274 00000002:00010000:2.0:1788572457.320976:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x5:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1788572457.320987:0:18435:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572457.321044:0:18435:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 1/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572457.321052:0:18435:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572457.321054:0:18435:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ada00/0x6d478a807bb115ec lrc: 0/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01684 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572457.321077:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.321085:0:18435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572457.321095:0:18435:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572457.321101:0:18435:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495386752/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572457.321111:0:18435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572457.321130:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495386752:192.168.202.102@tcp:101:lfs.500 00000100:00100000:2.0:1788572457.321304:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572457.321741:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495386496:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572457.323976:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495386752:192.168.202.102@tcp:101:lfs.500 00010000:00010000:2.0:1788572457.324009:0:18435:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324022:0:18435:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 4/1,0 mode: --/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324054:0:18435:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 4/1,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324067:0:18435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 4/1,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324082:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 5/1,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324090:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 5/1,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324097:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 4/0,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572457.324108:0:18435:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff93a249408000 x1875453495386752/t12884901956(12884901956) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00100000:2.0:1788572457.324121:0:18435:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408000 x1875453495386752/t12884901956(12884901956) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572457.324135:0:18435:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 3/0,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324149:0:18435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 2 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572457.324155:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x5:0x0] suppgids 500 500: rc 0 00000002:00100000:2.0:1788572457.324169:0:18435:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408000 x1875453495386752/t12884901956(12884901956) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00010000:2.0:1788572457.324182:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115f3 00000080:00010000:2.0:1788572457.324189:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115f3 from it ffffbaf6c1e83cc0 00010000:00010000:2.0:1788572457.324193:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 3/0,1 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.324200:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 2/0,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1788572457.324689:0:18435:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x5:0x0](0000000000000000) cmd=c00c66f3 arg=75e2a0 00000080:00200008:2.0:1788572457.325060:0:18435:0:(file.c:2483:do_file_read_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 0, count: 1048576 00000080:00200000:2.0:1788572457.325086:0:18435:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:2.0:1788572457.325109:0:18435:0:(file.c:2090:ll_file_io_generic()) f1e.sanity-pcc: read ppos: 0, bytes: 1048576 00000080:00200000:2.0:1788572457.325114:0:18435:0:(file.c:1848:ll_io_set_mirror()) f1e.sanity-pcc: desiginated mirror: 0 00000020:00200000:2.0:1788572457.325121:0:18435:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b7455e21[0x110, 1, [0x2000013a1:0x5:0x0] hash] 00000020:00200000:2.0:1788572457.325123:0:18435:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 1048576) 2048 0 00000080:00200000:2.0:1788572457.325129:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572457.325144:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 1, invalidate_lock 0 00800000:00000080:2.0:1788572457.325154:0:18435:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000002:00000080:2.0:1788572457.325162:0:18435:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00010000:00010000:2.0:1788572457.325196:0:18435:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572457.325213:0:18435:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1788572457.325275:0:18435:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572457.325283:0:18435:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572457.325308:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495387008:192.168.202.102@tcp:103:lfs.0 00000100:00100000:2.0:1788572457.326739:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572457.331211:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495387008:192.168.202.102@tcp:103:lfs.0 00010000:00010000:2.0:1788572457.331276:0:18435:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb1159f lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01534 expref: -99 pid: 18331 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 1 00000020:00100000:2.0:1788572457.331307:0:18435:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 58, max 7 00000100:00100000:2.0:1788572457.331317:0:18435:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940b800 x1875453495387136/t0(0) o58->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572457.331355:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495387136:192.168.202.102@tcp:58:lfs.0 00000100:00100000:2.0:1788572457.331439:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572457.342738:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495387136:192.168.202.102@tcp:58:lfs.0 00010000:00010000:2.0:1788572457.342787:0:18435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 31 res 8589939617/5 (0 0) 00000080:00010000:2.0:1788572457.342800:0:18435:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2000013a1:0x5:0x0](ffff93a251195348) 00000002:00010000:2.0:1788572457.342810:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x5:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572457.342856:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.342866:0:18435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.342876:0:18435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572457.342898:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495387264:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572457.342983:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572457.346296:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495387264:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572457.346343:0:18435:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.346357:0:18435:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572457.969587:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572457.969712:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518432512 00000100:00100000:1.0:1788572457.969722:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518432512 00000100:00100000:1.0:1788572457.969779:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 18 00000100:00100000:1.0:1788572457.969790:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18933:x1875453518432512:12345-192.168.202.102@tcp:107:hsm_cdtr.0 00000100:00100000:1.0:1788572457.969888:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18933:x1875453518432512:12345-192.168.202.102@tcp:107:hsm_cdtr.0 Request processed in 101us (311us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572457.969895:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 18 00000080:00200080:0.0:1788572457.971022:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c0086696 arg=7f3000000b60 00800000:00000080:0.0:1788572457.971058:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a2491b0000 uarg=0000000000000000 00000002:00000080:0.0:1788572457.971066:0:18436:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a2491b0000 uarg=0000000000000000 00000002:00000080:0.0:1788572457.971072:0:18436:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x2000013a1:0x5:0x0] from 18446744073709551615 #0 00000100:00100000:0.0:1788572457.971118:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387392:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:0.0:1788572457.971220:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572457.978976:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387392:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572457.979007:0:18436:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1e.sanity-pcc/f1e.sanity-pcc 00000080:00200080:2.0:1788572457.979174:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c01066f8 arg=6fb9e8 00800000:00000080:2.0:1788572457.979185:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c01066f8 len=16 karg=ffffbaf6c1e63cf0 uarg=ffffbaf6c1e63cc0 00000080:00200080:2.0:1788572457.979574:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8) cmd=c1406916 arg=7f3000000ce0 00000100:00100000:2.0:1788572457.979645:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387520:192.168.202.102@tcp:34:lhsmtool_posix.0 00000100:00100000:2.0:1788572457.979963:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572457.989706:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387520:192.168.202.102@tcp:34:lhsmtool_posix.0 00000080:00200000:2.0:1788572457.989791:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8), inode mode 4040 mask 1 00000080:00200000:2.0:1788572457.989810:0:18436:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=0x2000013a1:0x5:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110a1c8), file ffff93a246152180, open_flags 68800, mode 0 opened 0 00000080:00200000:2.0:1788572457.989816:0:18436:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=0x2000013a1:0x5:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110a1c8), intent=open 00000002:00010000:2.0:1788572457.989841:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x200000002:0x2:0x0], intent: open flags 041504001 00010000:00010000:2.0:1788572457.989911:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.989922:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572457.989950:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572457.989959:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940aa00 x1875453495387776/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572457.989968:0:18436:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572457.989995:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387776:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572457.990546:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572457.999430:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387776:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572457.999756:0:18436:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999772:0:18436:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999780:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999787:0:18436:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 3/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999796:0:18436:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999801:0:18436:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216200 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572457.999806:0:18436:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999835:0:18436:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999844:0:18436:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 5/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999852:0:18436:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572457.999861:0:18436:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 3/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572457.999868:0:18436:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb11601 lrc: 0/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572457.999890:0:18436:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a24940aa00 x1875453495387776/t12884901957(12884901957) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572457.999904:0:18436:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940aa00 x1875453495387776/t12884901957(12884901957) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572457.999919:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572457.999961:0:18436:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 0 00000080:00200000:2.0:1788572457.999965:0:18436:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00010000:2.0:1788572457.999970:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x0 00000080:00200008:2.0:1788572457.999989:0:18436:0:(file.c:1090:ll_file_open()) START file 0x2000013a1:0x5:0x0:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 1504000 00000002:00100000:2.0:1788572458.000008:0:18436:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940aa00 x1875453495387776/t12884901957(12884901957) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572458.000337:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 44 00000080:00200080:2.0:1788572458.000735:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c0086696 arg=7f3000000ce0 00800000:00000080:2.0:1788572458.000756:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a242c48000 uarg=0000000000000000 00000002:00000080:2.0:1788572458.000770:0:18436:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a242c48000 uarg=0000000000000000 00000002:00000080:2.0:1788572458.000776:0:18436:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x2000013a1:0x5:0x0] from 18446744073709551615 #0 00000100:00100000:2.0:1788572458.000825:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387904:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.000943:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.006509:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495387904:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572458.006538:0:18436:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1e.sanity-pcc/f1e.sanity-pcc 00000080:00200000:2.0:1788572458.006642:0:18436:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572458.006672:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572458.006690:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1788572458.006703:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:2.0:1788572458.006711:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572458.006717:0:18436:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572458.006721:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572458.006727:0:18436:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e63bf8 00010000:00010000:2.0:1788572458.006731:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1788572458.006739:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788572458.006755:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572458.006762:0:18436:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:2.0:1788572458.006780:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572458.006795:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 3 00000080:00200000:2.0:1788572458.006800:0:18436:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:5E621230:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), file ffff93a246153e00, open_flags 280c2, mode 8180 opened 0 00000080:00200000:2.0:1788572458.006806:0:18436:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:5E621230:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000002:00010000:2.0:1788572458.006825:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:5E621230:fd=09,[0x2000013a1:0x8:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open|creat flags 0500303 00010000:00010000:2.0:1788572458.006838:0:18436:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.006852:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.006892:0:18436:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.006899:0:18436:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.006902:0:18436:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be9a00/0x6d478a807bb115c9 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01606 expref: -99 pid: 18381 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.006928:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.006935:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572458.006945:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572458.006954:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408a80 x1875453495388160/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 680/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572458.006964:0:18436:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.006991:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388160:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.007134:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.022543:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388160:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572458.022576:0:18436:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 4/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022597:0:18436:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022607:0:18436:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022619:0:18436:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022630:0:18436:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022643:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022663:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022671:0:18436:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.022702:0:18436:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff93a249408a80 x1875453495388160/t12884901958(12884901958) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 680/664 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572458.022714:0:18436:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408a80 x1875453495388160/t12884901958(12884901958) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 680/664 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572458.022728:0:18436:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572458.022741:0:18436:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022753:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/8 (0 0) 00000080:00200000:2.0:1788572458.022759:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2000013a1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572458.022774:0:18436:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:5E621230:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000080:00200000:2.0:1788572458.022807:0:18436:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0x8:0x0](ffff93a251190908) 00000080:00200000:2.0:1788572458.022877:0:18436:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572458.022881:0:18436:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) 00000080:00010000:2.0:1788572458.022884:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) for lock 0x6d478a807bb11608 00000080:00010000:2.0:1788572458.022894:0:18436:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) 00000080:00200008:2.0:1788572458.022908:0:18436:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908), flags 500302 00000002:00100000:2.0:1788572458.022926:0:18436:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249408a80 x1875453495388160/t12884901958(12884901958) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572458.022959:0:18436:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11608 from it ffff93a251f0c120 00010000:00010000:2.0:1788572458.022963:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.022981:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572458.023006:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x8:0x0](ffff93a251190908), inode mode 8180 mask 40 00000080:00200000:2.0:1788572458.023051:0:18436:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572458.023066:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572458.023077:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788572458.023088:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788572458.023095:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572458.023100:0:18436:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572458.023103:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572458.023106:0:18436:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e63c10 00010000:00010000:2.0:1788572458.023110:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788572458.023117:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788572458.023125:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572458.023131:0:18436:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:2.0:1788572458.023135:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572458.023141:0:18436:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=. :VOLATILE:0000:5E621230:fd=09, flags=0 00000080:00200000:2.0:1788572458.023150:0:18436:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 3 00000080:00200000:2.0:1788572458.023155:0:18436:0:(namei.c:2351:ll_unlink()) VFS Op:name=. :VOLATILE:0000:5E621230:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8) 00010000:00010000:2.0:1788572458.023182:0:18436:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572458.023197:0:18436:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x8:0x0]: losing layout lock 00010000:00010000:2.0:1788572458.023388:0:18436:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.023395:0:18436:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.023398:0:18436:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11608 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c016e6 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572458.023440:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572458.023448:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408e00 x1875453495388416/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572458.023471:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388416:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.023614:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.031307:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388416:192.168.202.102@tcp:36:lhsmtool_posix.0 00000002:00010000:2.0:1788572458.031407:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0x8:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572458.031486:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 3/1,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.031496:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 3/1,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.031505:0:18436:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 3/1,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.031542:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388544:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.031626:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.038906:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388544:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572458.038938:0:18436:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.038954:0:18436:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.038966:0:18436:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.038981:0:18436:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff93a249408e00 x1875453495388544/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 568/1072 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572458.038992:0:18436:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.039006:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/8 (0 0) 00000080:00200000:2.0:1788572458.039013:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x2000013a1:0x8:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572458.039036:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) for lock 0x6d478a807bb1160f 00000080:00010000:2.0:1788572458.039044:0:18436:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1160f from it ffffbaf6c1e63c18 00010000:00010000:2.0:1788572458.039049:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.039073:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572458.039122:0:18436:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908) current size 0, valid attrs 1846, mode 93a2, uid 500, gid 500, new size 0, atime 1788572458.000000000, mtime 1788572458.000000000, ctime 1788572458.000000000 00000020:00100000:2.0:1788572458.039164:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572458.039170:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408e00 x1875453495388800/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572458.039189:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388800:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:0.0:1788572458.039753:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.052633:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388800:192.168.202.102@tcp:36:lhsmtool_posix.0 00010000:00010000:2.0:1788572458.052702:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 31 res 8589939617/8 (0 0) 00000080:00010000:2.0:1788572458.052714:0:18436:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2000013a1:0x8:0x0](ffff93a251190908) 00000002:00010000:2.0:1788572458.052725:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x8:0x0]) in obj [0x2000013a1:0x8:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572458.052770:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.052779:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.052789:0:18436:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.052808:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388928:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.052882:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.057995:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495388928:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572458.058034:0:18436:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.058087:0:18436:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.058103:0:18436:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a249409f80 x1875453495388928/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572458.058113:0:18436:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572458.058125:0:18436:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.058138:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/8 (0 0) 00000080:00200000:2.0:1788572458.058145:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2000013a1:0x8:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572458.058162:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) for lock 0x6d478a807bb11616 00000080:00010000:2.0:1788572458.058170:0:18436:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11616 from it ffffbaf6c1e63a68 00010000:00010000:2.0:1788572458.058174:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.058183:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.058202:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.058212:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572458.058398:0:18436:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x8:0x0](00000000f08d2d5e) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572458.058573:0:18436:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x8:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788572458.058581:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572458.058588:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572458.058608:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572458.058621:0:18436:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788572458.058901:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495389056:192.168.202.102@tcp:2:lhsmtool_posix.0 00000100:00100000:1.0:1788572458.063448:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495389056:192.168.202.102@tcp:2:lhsmtool_posix.0 00000080:00200000:0.0:1788572458.063510:0:18436:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572458.063528:0:18436:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908) current size 0, valid attrs 10001846, mode 93a2, uid 500, gid 500, new size 0, atime 1788572458.000000000, mtime 1788572458.000000000, ctime 1788572458.000000000, rc 0 00000080:00200008:0.0:1788572458.063588:0:18436:0:(file.c:479:ll_file_release()) START file 0x2000013a1:0x5:0x0:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 1504000 00000002:00100000:0.0:1788572458.063717:0:18436:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940aa00 x1875453495387776/t12884901957(12884901957) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572458.063729:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572458.063733:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d2d80 x1875453495389312/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572458.063753:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2d80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495389312:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:0.0:1788572458.063871:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.065684:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2d80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495389312:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:2.0:1788572458.065713:0:18436:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940aa00 x1875453495387776/t12884901957(12884901957) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572458.065771:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=400866d6 arg=7f3000000b78 00800000:00000080:2.0:1788572458.065782:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000002:00000080:2.0:1788572458.065788:0:18436:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000020:00100000:2.0:1788572458.065810:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:2.0:1788572458.065816:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940aa00 x1875453495389568/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572458.065836:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495389568:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.065897:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.072765:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495389568:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:2.0:1788572458.072962:0:18436:0:(file.c:6296:ll_getattr_dentry()) START file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572458.072970:0:18436:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x8:0x0](ffff93a251190908),name=. :VOLATILE:0000:5E621230:fd=09 00000002:00010000:2.0:1788572458.072987:0:18436:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x8:0x0]) in obj [0x2000013a1:0x8:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572458.073001:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 30 res 8589939617/8 (0 0) 00010000:00010000:2.0:1788572458.073049:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.073120:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.073131:0:18436:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18436 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.073152:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495389696:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.073307:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.077644:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495389696:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572458.077684:0:18436:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077696:0:18436:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077710:0:18436:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077722:0:18436:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077735:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077743:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077772:0:18436:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.077785:0:18436:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a24940aa00 x1875453495389696/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572458.077798:0:18436:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572458.077812:0:18436:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077827:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/8 (0 0) 00000080:00200000:2.0:1788572458.077834:0:18436:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x8:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572458.077894:0:18436:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x8:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788572458.077901:0:18436:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) for lock 0x6d478a807bb1161d 00000080:00010000:2.0:1788572458.077907:0:18436:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x8:0x0](ffff93a251190908) 00000080:00010000:2.0:1788572458.077910:0:18436:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1161d from it ffffbaf6c1e63cd8 00010000:00010000:2.0:1788572458.077914:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.077924:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572458.077963:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572458.077973:0:18436:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0x8:0x0] 00000008:00010000:2.0:1788572458.077993:0:18436:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572458.078004:0:18436:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010430, osc lock ffff93a2489c7db0, flags 0x400001002 00010000:00010000:2.0:1788572458.078066:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.078076:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572458.078115:0:18436:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788572458.078346:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495389824:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:3.0:1788572458.080927:0:2355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.080955:0:2355:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.080970:0:2355:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.080982:0:2355:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788572458.080997:0:2355:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788572458.081009:0:2355:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788572458.081026:0:2355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.081034:0:2355:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.081057:0:2355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572458.081066:0:2355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788572458.081088:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495389824:192.168.202.102@tcp:101:lhsmtool_posix.0 00000080:00200000:2.0:1788572458.081187:0:18436:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x8:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788572458.081193:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba1bc0 (ffff93a2516c0c68/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.081197:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba1bc0 (ffff93a2516c0c68/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.081201:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010430 (ffff93a2516c0b48/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.081204:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba1bc0 (ffff93a2516c0c68/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.081207:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010430 (ffff93a2516c0c68/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.081218:0:18436:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1788572458.081315:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=401866d8 arg=7f3007b90910 00800000:00000080:2.0:1788572458.081325:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000002:00000080:2.0:1788572458.081333:0:18436:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000020:00100000:2.0:1788572458.081369:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:2.0:1788572458.081377:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409f80 x1875453495390080/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572458.081401:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390080:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.081525:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.085812:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390080:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:2.0:1788572458.086029:0:18436:0:(file.c:2650:do_file_write_iter()) START file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788572458.086045:0:18436:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788572458.086071:0:18436:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788572458.086078:0:18436:0:(file.c:2090:ll_file_io_generic()) . :VOLATILE:0000:5E621230:fd=09: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788572458.086084:0:18436:0:(file.c:1848:ll_io_set_mirror()) . :VOLATILE:0000:5E621230:fd=09: desiginated mirror: 0 00000020:00200000:2.0:1788572458.086092:0:18436:0:(cl_io.c:214:cl_io_rw_init()) header@00000000ad2c6530[0x110, 1, [0x2000013a1:0x8:0x0] hash] 00000020:00200000:2.0:1788572458.086098:0:18436:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788572458.086104:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572458.086120:0:18436:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788572458.086128:0:18436:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788572458.086132:0:18436:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572458.086149:0:18436:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788572458.086163:0:18436:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572458.086168:0:18436:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010430, osc lock ffff93a2489c7db0, flags 0x0 00010000:00010000:2.0:1788572458.086185:0:18436:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a2495c1400 type 11 mode 2 res 10737419264/67 (0 4095) 00010000:00010000:2.0:1788572458.086260:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 3/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.086273:0:18436:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 3/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.086285:0:18436:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 3/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.086305:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390208:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.086404:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.090736:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572458.090981:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518432640 00000100:00100000:1.0:1788572458.090993:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518432640 00000100:00100000:1.0:1788572458.091064:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 19 00000100:00100000:1.0:1788572458.091076:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:19590:x1875453518432640:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572458.091096:0:16828:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572458.091189:0:16828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572458.091196:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:19590:x1875453518432640:12345-192.168.202.102@tcp:104: Request processed in 123us (470us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572458.091203:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 19 00010000:00010000:2.0:1788572458.091765:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572458.091775:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.091786:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216a00 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:2.0:1788572458.091792:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572458.091860:0:17521:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572458.091874:0:17521:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572458.091923:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.091931:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.091962:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.091971:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 0000000016d7c723/0x6d478a807bb11624 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0172c expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.091986:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.091987:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.091989:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572458.092173:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495390336:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572458.094541:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390208:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572458.094574:0:18436:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.094588:0:18436:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.094596:0:18436:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572458.095522:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572458.095846:0:16828:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1788572458.095862:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518432768 00000100:00100000:1.0:1788572458.095868:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518432768 00000100:00100000:1.0:1788572458.095912:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 20 00000100:00100000:1.0:1788572458.095919:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a27429cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:18886:x1875453518432768:12345-192.168.202.102@tcp:105: 00010000:00010000:1.0:1788572458.095930:0:16828:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.095939:0:16828:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.095945:0:16828:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.095961:0:16828:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.095968:0:16828:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.095977:0:16828:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.095992:0:16828:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff93a243216000) 00000100:00100000:1.0:1788572458.096036:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a27429cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:18886:x1875453518432768:12345-192.168.202.102@tcp:105: Request processed in 117us (519us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572458.096041:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 20 00010000:00010000:2.0:1788572458.096373:0:18436:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.096381:0:18436:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.096392:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572458.096401:0:18436:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.096410:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.096415:0:18436:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572458.096457:0:18436:0:(vvp_io.c:1292:vvp_io_write_start()) . :VOLATILE:0000:5E621230:fd=09: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788572458.096691:0:18436:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788572458.096739:0:18436:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788572458.096748:0:18436:0:(rw26.c:929:ll_write_end()) page@00000000daf7067c[3 000000002447d7a5 2 1 00000000e4a11e27 0 0 0] 00000080:00200000:2.0:1788572458.096752:0:18436:0:(rw26.c:929:ll_write_end()) vmpage @00000000f270983f 17ffffc0001001 3:0 ffff93a27425ab60 0 lru 00000080:00200000:2.0:1788572458.096759:0:18436:0:(rw26.c:929:ll_write_end()) osc-page@00000000a97cda52 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000d3afed6b 00000000461fcaab > 3< 0 0 > 4< 0 0 8 18743296 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572458.096761:0:18436:0:(rw26.c:929:ll_write_end()) end page@00000000daf7067c 00000080:00200000:2.0:1788572458.096769:0:18436:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788572458.096773:0:18436:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788572458.096793:0:18436:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000ec3063e0 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572458.096811:0:18436:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff93a251190ab8, count 1, dirtied 1 00000080:00200000:2.0:1788572458.096815:0:18436:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788572458.096818:0:18436:0:(vvp_io.c:1358:vvp_io_write_start()) . :VOLATILE:0000:5E621230:fd=09: write bytes 0, result: 1024 00000020:00010000:2.0:1788572458.099195:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba57b8 (ffff93a2516c0c68/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.099209:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba57b8 (ffff93a2516c0c68/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.099214:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010430 (ffff93a2516c0b48/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572458.099226:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.099259:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572458.099278:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba57b8 (ffff93a2516c0c68/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.099281:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010430 (ffff93a2516c0c68/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.099291:0:18436:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788572458.099297:0:18436:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788572458.099304:0:18436:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572458.099308:0:18436:0:(file.c:2238:ll_file_io_generic()) . :VOLATILE:0000:5E621230:fd=09: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788572458.099319:0:18436:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x8:0x0] updating i_size 1024 i_blocks 1 00000080:00200000:2.0:1788572458.099323:0:18436:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788572458.099632:0:18436:0:(file.c:5384:ll_fsync()) START file: name=. :VOLATILE:0000:5E621230:fd=09, fid=[0x2000013a1:0x8:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/1 00000080:00200000:2.0:1788572458.099647:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572458.099656:0:18436:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788572458.099676:0:18436:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000ec3063e0 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572458.099702:0:18436:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000ec3063e0 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.099745:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00100000:2.0:1788572458.099823:0:18436:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff93a249409f80 - object 0x280000400:67 offset 0<>1024 00000100:00100000:3.0:1788572458.099910:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495390464:192.168.202.102@tcp:4:lhsmtool_posix.0 00000008:00010000:3.0:1788572458.117707:0:2355:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000ec3063e0 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00010000:3.0:1788572458.117779:0:2355:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000ec3063e0 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00100000:3.0:1788572458.117810:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495390464:192.168.202.102@tcp:4:lhsmtool_posix.0 00000080:00200000:0.0:1788572458.117953:0:18436:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1788572458.118023:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390592:192.168.202.102@tcp:44:lhsmtool_posix.0 00000100:00100000:0.0:1788572458.118128:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.119789:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495390336:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572458.172064:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390592:192.168.202.102@tcp:44:lhsmtool_posix.0 00000080:00200000:0.0:1788572458.172108:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572458.172123:0:18436:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788572458.172221:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495390720:192.168.202.102@tcp:16:lhsmtool_posix.0 00000100:00100000:0.0:1788572458.213143:0:2353:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244a1e800: committing for last_committed 12884901903 gen 1 00000100:00100000:0.0:1788572458.213257:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495390720:192.168.202.102@tcp:16:lhsmtool_posix.0 00000080:00200000:2.0:1788572458.213723:0:18436:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x8:0x0] updating i_size 1024 i_blocks 11 00000080:00200000:2.0:1788572458.213740:0:18436:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1788572458.213779:0:18436:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x8:0x0](ffff93a251190908) cmd=c03066b0 arg=7f3007b8fac0 00000080:00200008:2.0:1788572458.213790:0:18436:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908) current size 1024, valid attrs 100001f0, mode 0, uid 0, gid 0, new size 0, atime 1788572455.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000 00010000:00010000:2.0:1788572458.213823:0:18436:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.213849:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.213892:0:18436:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.213899:0:18436:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.213902:0:18436:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217c00/0x6d478a807bb1161d lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01725 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572458.213938:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572458.213946:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940b100 x1875453495390976/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572458.213968:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390976:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.214075:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.218439:0:18436:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901960 gen 1 00000100:00100000:2.0:1788572458.218541:0:18436:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a2460d2300 x1875453495386368/t12884901955(12884901955) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000100:00100000:2.0:1788572458.218553:0:18436:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a249408000 x1875453495386752/t12884901956(12884901956) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000100:00100000:2.0:1788572458.218562:0:18436:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a249408a80 x1875453495388160/t12884901958(12884901958) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572458.218576:0:18436:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff93a24940b100 x1875453495390976/t12884901961(12884901961) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572474 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572458.218588:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495390976:192.168.202.102@tcp:36:lhsmtool_posix.0 00000080:00200000:2.0:1788572458.218798:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572458.218812:0:18436:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572458.218825:0:18436:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788572458.218842:0:18436:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572458.218846:0:18436:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010430, osc lock ffff93a2489c7db0, flags 0x0 00010000:00010000:2.0:1788572458.218868:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572458.218903:0:18436:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572458.218917:0:18436:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572458.218927:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572458.218934:0:18436:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00100000:3.0:1788572458.219038:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495391104:192.168.202.102@tcp:2:lhsmtool_posix.0 00000100:00100000:3.0:1788572458.222901:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495391104:192.168.202.102@tcp:2:lhsmtool_posix.0 00000020:00010000:2.0:1788572458.222960:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba57b8 (ffff93a2516c0c68/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.222964:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba57b8 (ffff93a2516c0c68/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.222969:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010430 (ffff93a2516c0b48/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572458.222979:0:18436:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572458.222995:0:18436:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:2.0:1788572458.223008:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba57b8 (ffff93a2516c0c68/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.223010:0:18436:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010430 (ffff93a2516c0c68/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.223025:0:18436:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00800000:00000080:2.0:1788572458.223061:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d4 len=392 karg=ffff93a245518000 uarg=0000000000000000 00000002:00000080:2.0:1788572458.223073:0:18436:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d4 len=392 karg=ffff93a245518000 uarg=0000000000000000 00000020:00100000:2.0:1788572458.223103:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 55, max 7 00000100:00100000:2.0:1788572458.223111:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409c00 x1875453495391360/t0(0) o55->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 472/224 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572458.223135:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495391360:192.168.202.102@tcp:55:lhsmtool_posix.0 00000100:00100000:2.0:1788572458.223280:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.225626:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572458.225783:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518433024 00000100:00100000:1.0:1788572458.225792:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518433024 00000100:00100000:1.0:1788572458.225877:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 21 00000100:00100000:1.0:1788572458.225888:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18901:x1875453518433024:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572458.225909:0:16828:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572458.226028:0:16828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572458.226034:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18901:x1875453518433024:12345-192.168.202.102@tcp:104: Request processed in 149us (420us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572458.226057:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 21 00010000:00010000:2.0:1788572458.227199:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572458.227208:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.227219:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216200 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572458.227223:0:16838:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.227246:0:16838:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.227256:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.227313:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.227320:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572458.227392:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495391488:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572458.229898:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.229940:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216200/0x6d478a807bb1160f lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x8:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01709 expref: -99 pid: 18436 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.229961:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.229963:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.229966:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572458.230906:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495391360:192.168.202.102@tcp:55:lhsmtool_posix.0 00000080:00200008:2.0:1788572458.230943:0:18436:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908) current size 1024, valid attrs 100001f0, mode 0, uid 0, gid 0, new size 0, atime 1788572455.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000, rc 0 00000080:00200080:2.0:1788572458.230962:0:18436:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=400866d7 arg=7f3000000b78 00000080:00200000:2.0:1788572458.230981:0:18436:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572458.230994:0:18436:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572458.231066:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495391616:192.168.202.102@tcp:1:lhsmtool_posix.0 00000100:00100000:3.0:1788572458.233303:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495391488:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572458.233740:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572458.233840:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518433152 00000100:00100000:1.0:1788572458.233848:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518433152 00000100:00100000:1.0:1788572458.233943:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 22 00000100:00100000:1.0:1788572458.233954:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19590:x1875453518433152:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572458.234109:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572458.234261:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572458.234268:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19590:x1875453518433152:12345-192.168.202.102@tcp:104: Request processed in 316us (530us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572458.234277:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 22 00010000:00010000:3.0:1788572458.234829:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572458.234837:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572458.234847:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216000 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:3.0:1788572458.234853:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400020000 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572458.234877:0:16837:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572458.236123:0:16837:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572458.236183:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 2/0,0 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572458.236193:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.236220:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 1/0,0 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572458.236241:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000f33bfc5f/0x6d478a807bb1162b lrc: 0/0,0 mode: --/PW res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c01733 expref: -99 pid: 18436 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572458.236276:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.236278:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.236279:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572458.236291:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572458.236307:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495391744:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572458.239165:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495391744:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572458.239254:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495391616:192.168.202.102@tcp:1:lhsmtool_posix.0 00800000:00000080:0.0:1788572458.239459:0:18436:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000002:00000080:0.0:1788572458.239527:0:18436:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000020:00100000:0.0:1788572458.239557:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:0.0:1788572458.239565:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d1500 x1875453495392000/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572458.239588:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495392000:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:0.0:1788572458.239746:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572458.242899:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572458.243001:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518433280 00000100:00100000:0.0:1788572458.243012:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518433280 00000100:00100000:0.0:1788572458.243100:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 23 00000100:00100000:0.0:1788572458.243111:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518433280:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572458.243144:0:16828:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572458.243290:0:16828:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572458.243298:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518433280:12345-192.168.202.102@tcp:104: Request processed in 189us (409us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572458.243306:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 23 00010000:00010000:2.0:1788572458.244576:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572458.244590:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.244603:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217400 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572458.244608:0:17521:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.244618:0:17521:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.244627:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 3/0,0 mode: CR/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572458.244661:0:17521:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x8:0x0]: losing layout lock 00010000:00010000:2.0:1788572458.244712:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 2/0,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.244736:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.244772:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.244780:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11616 lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x8:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01717 expref: -99 pid: 18436 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.244802:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.244804:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.244805:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572458.244929:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495392128:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572458.249928:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495392128:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572458.256776:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572458.257027:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518433664 00000100:00100000:0.0:1788572458.257040:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518433664 00000100:00100000:0.0:1788572458.257100:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 24 00000100:00100000:0.0:1788572458.257111:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518433664:12345-192.168.202.102@tcp:105: 00010000:00010000:0.0:1788572458.257130:0:16827:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 5/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.257142:0:16827:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 5/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.257151:0:16827:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 5/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.257161:0:16827:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 5/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.257171:0:16827:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 6/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.257181:0:16827:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 6/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x80000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.257225:0:16827:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff93a243217200) 00000100:00100000:0.0:1788572458.257322:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18900:x1875453518433664:12345-192.168.202.102@tcp:105: Request processed in 212us (555us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572458.257329:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 24 00010000:00010000:2.0:1788572458.257403:0:18435:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.257414:0:18435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.257441:0:18435:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a24940b800 x1875453495387264/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 376/344 e 0 to 0 dl 1788572473 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572458.257451:0:18435:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.257472:0:18435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572458.257480:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2000013a1:0x5:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572458.257499:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb115fa 00000080:00010000:2.0:1788572458.257506:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb115fa from it ffffbaf6c1e83810 00010000:00010000:2.0:1788572458.257510:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.257532:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.257553:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.257564:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572458.257572:0:18435:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572458.257621:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495392256:192.168.202.102@tcp:49:lfs.0 00000100:00100000:2.0:1788572458.257670:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.260982:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495392000:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:0.0:1788572458.261123:0:18436:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:5E621230:fd=09:[0x2000013a1:0x8:0x0](ffff93a251190908), flags 500002 00000002:00100000:0.0:1788572458.261183:0:18436:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249408a80 x1875453495388160/t12884901958(12884901958) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572458.261195:0:18436:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572458.261200:0:18436:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d2a00 x1875453495392512/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572458.261222:0:18436:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495392512:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:0.0:1788572458.261338:0:18436:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.264548:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495392256:192.168.202.102@tcp:49:lfs.0 00010000:00010000:2.0:1788572458.264617:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.264635:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572458.264661:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572458.264670:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572458.264724:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.264752:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572458.264762:0:18435:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572458.264782:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572458.264787:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572458.264827:0:18435:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4294967294 -> 2 00010000:00010000:2.0:1788572458.264833:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.264841:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572458.264850:0:18435:0:(vvp_io.c:351:vvp_io_fini()) [0x2000013a1:0x5:0x0] layout changed from 1 to 2. 00000080:00200000:2.0:1788572458.264854:0:18435:0:(file.c:2238:ll_file_io_generic()) f1e.sanity-pcc: 1 io complete with rc: -61, result: 0, restart: 1 00000080:00200000:2.0:1788572458.264858:0:18435:0:(file.c:2245:ll_file_io_generic()) f1e.sanity-pcc: restart read from ppos=0 bytes=1048576 retries=999 ret=0: rc = -61 00000080:00200000:2.0:1788572458.264865:0:18435:0:(file.c:1848:ll_io_set_mirror()) f1e.sanity-pcc: desiginated mirror: 0 00000020:00200000:2.0:1788572458.264872:0:18435:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b7455e21[0x110, 1, [0x2000013a1:0x5:0x0] hash] 00000020:00200000:2.0:1788572458.264875:0:18435:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 1048576) 2048 0 00000080:00200000:2.0:1788572458.264877:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1788572458.264892:0:18435:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1048576) 1048576, 1048576 00020000:00200000:2.0:1788572458.264896:0:18435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572458.264913:0:18435:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 255] 00000008:00010000:2.0:1788572458.264927:0:18435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 1 00000008:00010000:2.0:1788572458.264933:0:18435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010fb0, osc lock ffff93a2489c7db0, flags 0x40000 00010000:00010000:2.0:1788572458.264984:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.265001:0:18435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x40000 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.265013:0:18435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.265035:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495392640:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572458.265139:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.269437:0:18436:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18436:1875453495392512:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:0.0:1788572458.269481:0:18436:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249408a80 x1875453495388160/t12884901958(12884901958) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572458.272005:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495392640:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572458.272034:0:18435:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272054:0:18435:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272068:0:18435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272083:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572458.272096:0:18435:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 1024, setting rss=1024; kms=1024, end=18446744073709551615 ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272133:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272142:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572458.272169:0:18435:0:(vvp_io.c:835:vvp_io_read_start()) f1e.sanity-pcc: read [0, 1048576) 00000001:00010000:2.0:1788572458.272194:0:18435:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0x5:0x0] 00000008:00010000:2.0:1788572458.272205:0:18435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572458.272209:0:18435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010430, osc lock ffff93a2489c6130, flags 0x400001002 00010000:00010000:2.0:1788572458.272226:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272273:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572458.272283:0:18435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572458.272286:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 7/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.272292:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572458.272299:0:18435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572458.272306:0:18435:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x5:0x0] updating i_size 1024 i_blocks 11 00000020:00010000:2.0:1788572458.272310:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.272320:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.272323:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010430 (ffff93a2516c02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.272327:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.272330:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010430 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.272341:0:18435:0:(vvp_io.c:893:vvp_io_read_start()) count:1048576 iocb pos:0 00000008:00010000:2.0:1788572458.272417:0:18435:0:(osc_cache.c:3059:osc_queue_sync_pages()) ### no dlm lock: extent: ffff93a24972ab40 00000008:00010000:2.0:1788572458.272461:0:138:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff93a24972ab40 00010000:00010000:2.0:1788572458.272495:0:138:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00100000:2.0:1788572458.272528:0:138:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff93a24940b800 - object 0x280000400:67 offset 0<>4096 00000100:00100000:3.0:1788572458.272667:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572458.272687:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495392768:192.168.202.102@tcp:3:lfs.0 00000008:00010000:3.0:1788572458.276163:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff93a24972ab40 00000020:00200000:3.0:1788572458.276181:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1788572458.276201:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff93a24972ab40 00000100:00100000:3.0:1788572458.276263:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495392768:192.168.202.102@tcp:3:lfs.0 00000020:00010000:2.0:1788572458.276503:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.276508:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.276512:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010fb0 (ffff93a2516c02d8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572458.276523:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.276541:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572458.276559:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.276562:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010fb0 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.276570:0:18435:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788572458.276581:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572458.276585:0:18435:0:(file.c:2238:ll_file_io_generic()) f1e.sanity-pcc: 1 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788572458.276591:0:18435:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 1024 00000080:00200008:2.0:1788572458.276618:0:18435:0:(file.c:2483:do_file_read_iter()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0], ppos: 1024, count: 1048576 00000080:00200000:2.0:1788572458.276627:0:18435:0:(file.c:2090:ll_file_io_generic()) f1e.sanity-pcc: read ppos: 1024, bytes: 1048576 00000080:00200000:2.0:1788572458.276647:0:18435:0:(file.c:1848:ll_io_set_mirror()) f1e.sanity-pcc: desiginated mirror: 0 00000020:00200000:2.0:1788572458.276653:0:18435:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b7455e21[0x110, 2, [0x2000013a1:0x5:0x0] hash] 00000020:00200000:2.0:1788572458.276655:0:18435:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [1024, 1049600) 2048 0 00000080:00200000:2.0:1788572458.276659:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00020000:00200000:2.0:1788572458.276677:0:18435:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 1024 chunk: [1024, 1049600) 1049600, 1048576 00020000:00200000:2.0:1788572458.276681:0:18435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572458.276696:0:18435:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 256] 00000008:00010000:2.0:1788572458.276706:0:18435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 1 00000008:00010000:2.0:1788572458.276711:0:18435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010fb0, osc lock ffff93a2489c7db0, flags 0x40000 00010000:00010000:2.0:1788572458.276731:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276743:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 1052671) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276755:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276763:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276770:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572458.276779:0:18435:0:(vvp_io.c:835:vvp_io_read_start()) f1e.sanity-pcc: read [1024, 1049600) 00000001:00010000:2.0:1788572458.276794:0:18435:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0x5:0x0] 00000008:00010000:2.0:1788572458.276800:0:18435:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572458.276802:0:18435:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010430, osc lock ffff93a2489c6130, flags 0x400001002 00010000:00010000:2.0:1788572458.276809:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276828:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572458.276836:0:18435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572458.276839:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 7/2,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276846:0:18435:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572458.276852:0:18435:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572458.276858:0:18435:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0x5:0x0] updating i_size 1024 i_blocks 11 00000020:00010000:2.0:1788572458.276861:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.276863:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.276865:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010430 (ffff93a2516c02d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.276868:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.276870:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010430 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.276875:0:18435:0:(vvp_io.c:893:vvp_io_read_start()) count:1048576 iocb pos:1024 00000020:00010000:2.0:1788572458.276879:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:2.0:1788572458.276881:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572458.276883:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010fb0 (ffff93a2516c02d8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572458.276887:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.276894:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788572458.276908:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5038 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572458.276910:0:18435:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010fb0 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572458.276914:0:18435:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:2.0:1788572458.276918:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572458.276921:0:18435:0:(file.c:2238:ll_file_io_generic()) f1e.sanity-pcc: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:2.0:1788572458.276923:0:18435:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200000:2.0:1788572458.277040:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00020000:00200000:2.0:1788572458.277045:0:18435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572458.277120:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572458.277136:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495392896:192.168.202.102@tcp:1:lfs.500 00000100:00100000:1.0:1788572458.280023:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572458.280116:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518433792 00000100:00100000:0.0:1788572458.280126:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518433792 00000100:00100000:0.0:1788572458.280181:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 25 00000100:00100000:0.0:1788572458.280193:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19591:x1875453518433792:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572458.280211:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572458.280343:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572458.280360:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19591:x1875453518433792:12345-192.168.202.102@tcp:104: Request processed in 170us (347us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572458.280367:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 25 00010000:00010000:2.0:1788572458.280920:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572458.280929:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.280939:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217400 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:2.0:1788572458.280945:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800428400000000 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572458.280963:0:16838:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572458.281030:0:16838:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572458.281085:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.281093:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.281124:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.281144:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff93a244a1e800 lock: 00000000c017a135/0x6d478a807bb11632 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0176b expref: -99 pid: 18435 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.281167:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.281169:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.281170:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572458.281216:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572458.281268:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495393024:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572458.284043:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495393024:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572458.286018:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495392896:192.168.202.102@tcp:1:lfs.500 00010000:00010000:2.0:1788572458.286138:0:18435:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 2/0,0 mode: PW/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.286157:0:18435:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216800/0x6d478a807bb115f3 lrc: 0/0,0 mode: --/PW res: [0x2000013a1:0x5:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0x977d9e7969c01699 expref: -99 pid: 18435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.286177:0:18435:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249408000 x1875453495386752/t12884901956(12884901956) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572458.286189:0:18435:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572458.286194:0:18435:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940ad80 x1875453495393280/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572458.286215:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495393280:192.168.202.102@tcp:35:lfs.500 00000100:00100000:2.0:1788572458.286591:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572458.290477:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572458.290562:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518433920 00000100:00100000:0.0:1788572458.290572:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518433920 00000100:00100000:0.0:1788572458.290630:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 26 00000100:00100000:0.0:1788572458.290641:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18904:x1875453518433920:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572458.290669:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572458.290743:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572458.290750:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18904:x1875453518433920:12345-192.168.202.102@tcp:104: Request processed in 111us (295us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572458.290757:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 26 00010000:00010000:2.0:1788572458.290806:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572458.290813:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.290822:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217200 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572458.290826:0:16837:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.290849:0:16837:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.290857:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 3/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572458.290879:0:16837:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1788572458.290918:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 2/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.290924:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.290948:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.290956:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb115fa lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c016ae expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572458.290973:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.290975:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.290976:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572458.291075:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495393408:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572458.295182:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495393408:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572458.299372:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495393280:192.168.202.102@tcp:35:lfs.500 00000002:00100000:2.0:1788572458.299408:0:18435:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249408000 x1875453495386752/t12884901956(12884901956) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00010000:2.0:1788572458.299476:0:18435:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2000013a1:0x5:0x0](ffff93a251195348) 00000002:00010000:2.0:1788572458.299487:0:18435:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x5:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572458.299541:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.299571:0:18435:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.299582:0:18435:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18435 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.299604:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495393536:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572458.299691:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.302008:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495393536:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572458.302041:0:18435:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.302056:0:18435:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.302072:0:18435:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a24940ad80 x1875453495393536/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572458.302084:0:18435:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572458.302097:0:18435:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.302111:0:18435:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572458.302117:0:18435:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2000013a1:0x5:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572458.302134:0:18435:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb11639 00000080:00010000:2.0:1788572458.302143:0:18435:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11639 from it ffffbaf6c1e83b70 00010000:00010000:2.0:1788572458.302160:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.302170:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.302190:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.302199:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572458.302207:0:18435:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.302256:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572458.302264:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572458.302278:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572458.302290:0:18435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572458.302317:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572458.302328:0:18435:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.302335:0:18435:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572458.302354:0:18435:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572458.302368:0:18435:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572458.302372:0:18435:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572458.302380:0:18435:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572458.302409:0:18435:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4294967294 -> 3 00010000:00010000:2.0:1788572458.302414:0:18435:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.302432:0:18435:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788572458.302493:0:18435:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 104002 00000002:00100000:2.0:1788572458.302530:0:18435:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a2460d2300 x1875453495386368/t12884901955(12884901955) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572458.302541:0:18435:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572458.302547:0:18435:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940ad80 x1875453495393792/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572458.302568:0:18435:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495393792:192.168.202.102@tcp:35:lfs.500 00000100:00100000:2.0:1788572458.302680:0:18435:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.305202:0:18435:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18435:1875453495393792:192.168.202.102@tcp:35:lfs.500 00000002:00100000:2.0:1788572458.305251:0:18435:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a2460d2300 x1875453495386368/t12884901955(12884901955) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572473 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00010000:00010000:2.0:1788572458.336629:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572458.392536:0:18447:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572458.392562:0:18447:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572458.392581:0:18447:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788572458.392596:0:18447:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788572458.392604:0:18447:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572458.392612:0:18447:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572458.392615:0:18447:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572458.392623:0:18447:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e93bf8 00010000:00010000:2.0:1788572458.392628:0:18447:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1788572458.392636:0:18447:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788572458.392651:0:18447:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572458.392659:0:18447:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=267 00000080:00200000:2.0:1788572458.392665:0:18447:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 44 00000080:00200008:2.0:1788572458.392671:0:18447:0:(file.c:1090:ll_file_open()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00010000:2.0:1788572458.392685:0:18447:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:2.0:1788572458.392733:0:18447:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.392750:0:18447:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572458.392763:0:18447:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572458.392773:0:18447:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940ad80 x1875453495394048/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572458.392782:0:18447:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572458.392803:0:18447:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18447:1875453495394048:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572458.392993:0:18447:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.401034:0:18447:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18447:1875453495394048:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572458.401059:0:18447:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401074:0:18447:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401082:0:18447:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401089:0:18447:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401099:0:18447:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401105:0:18447:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217400 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572458.401126:0:18447:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401133:0:18447:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401141:0:18447:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401148:0:18447:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572458.401155:0:18447:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572458.401162:0:18447:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11640 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18447 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572458.401183:0:18447:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a24940ad80 x1875453495394048/t12884901978(12884901978) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572458.401196:0:18447:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940ad80 x1875453495394048/t12884901978(12884901978) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572458.401209:0:18447:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572458.401244:0:18447:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000002:00100000:2.0:1788572458.401261:0:18447:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940ad80 x1875453495394048/t12884901978(12884901978) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572458.401352:0:18447:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) cmd=901066fc arg=7fff5819c8c0 00000100:00100000:2.0:1788572458.401411:0:18447:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18447:1875453495394176:192.168.202.102@tcp:34:lfs.0 00000100:00100000:2.0:1788572458.402635:0:18447:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.409862:0:18447:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18447:1875453495394176:192.168.202.102@tcp:34:lfs.0 00000080:00200008:2.0:1788572458.409967:0:18447:0:(file.c:479:ll_file_release()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00100000:2.0:1788572458.410031:0:18447:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940ad80 x1875453495394048/t12884901978(12884901978) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572474 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572458.410044:0:18447:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572458.410049:0:18447:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495394432/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572458.410071:0:18447:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18447:1875453495394432:192.168.202.102@tcp:35:lfs.0 00000100:00100000:2.0:1788572458.410153:0:18447:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572458.413797:0:18447:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18447:1875453495394432:192.168.202.102@tcp:35:lfs.0 00000002:00100000:2.0:1788572458.413837:0:18447:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940ad80 x1875453495394048/t12884901978(12884901978) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788572458.522140:0:18453:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572458.522154:0:18453:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572458.522173:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.522191:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:0.0:1788572458.522207:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1788572458.522215:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:0.0:1788572458.522223:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572458.522301:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572458.522309:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e4bcb8 00010000:00010000:0.0:1788572458.522314:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:0.0:1788572458.522323:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1788572458.522350:0:18453:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572458.522357:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.522364:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1788572458.522373:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1788572458.522380:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:0.0:1788572458.522384:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572458.522387:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572458.522390:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e4bb58 00010000:00010000:0.0:1788572458.522393:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1788572458.522400:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:0.0:1788572458.522409:0:18453:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572458.522417:0:18453:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=8 00000080:00200008:0.0:1788572458.522439:0:18453:0:(file.c:6296:ll_getattr_dentry()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572458.522443:0:18453:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8),name=d1e.sanity-pcc 00000002:00010000:0.0:1788572458.522455:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.522461:0:18453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 30 res 8589939617/4 (0 0) 00010000:00010000:0.0:1788572458.522503:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.522510:0:18453:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.522520:0:18453:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572458.522546:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495394560:192.168.202.102@tcp:101:lfs.500 00000100:00100000:0.0:1788572458.522776:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.533471:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495394560:192.168.202.102@tcp:101:lfs.500 00010000:00010000:0.0:1788572458.533504:0:18453:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533523:0:18453:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533532:0:18453:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533543:0:18453:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533556:0:18453:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533586:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533594:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533602:0:18453:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572458.533615:0:18453:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a2460d1c00 x1875453495394560/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:0.0:1788572458.533627:0:18453:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533638:0:18453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/4 (0 0) 00000080:00200000:0.0:1788572458.533645:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:0.0:1788572458.533668:0:18453:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00010000:0.0:1788572458.533684:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb11647 00000080:00010000:0.0:1788572458.533691:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000080:00010000:0.0:1788572458.533695:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11647 from it ffffbaf6c1e4bcb8 00010000:00010000:0.0:1788572458.533698:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533707:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572458.533778:0:18453:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572458.533794:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572458.533804:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:0.0:1788572458.533815:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:0.0:1788572458.533833:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:0.0:1788572458.533837:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572458.533840:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572458.533844:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e4bb58 00010000:00010000:0.0:1788572458.533847:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:0.0:1788572458.533855:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:0.0:1788572458.533863:0:18453:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572458.533870:0:18453:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:0.0:1788572458.533874:0:18453:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:0.0:1788572458.533891:0:18453:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags=8 00000080:00200000:0.0:1788572458.533903:0:18453:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=getattr 00000002:00010000:0.0:1788572458.533914:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572458.533948:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533965:0:18453:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.533975:0:18453:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572458.533998:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495394688:192.168.202.102@tcp:101:lfs.500 00000100:00100000:0.0:1788572458.534084:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572458.537606:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495394688:192.168.202.102@tcp:101:lfs.500 00010000:00010000:0.0:1788572458.537638:0:18453:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537656:0:18453:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537768:0:18453:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537780:0:18453:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537791:0:18453:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537804:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537887:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572458.537896:0:18453:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572458.537909:0:18453:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a2460d1c00 x1875453495394688/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:0.0:1788572458.537921:0:18453:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572458.538013:0:18453:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.538025:0:18453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:1.0:1788572458.538032:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:1.0:1788572458.538074:0:18453:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 1024 00000080:00200000:1.0:1788572458.538078:0:18453:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:1.0:1788572458.538091:0:18453:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 3 -> 3 00000080:00010000:1.0:1788572458.538095:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1164e 00000080:00010000:1.0:1788572458.538105:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:1.0:1788572458.538109:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1164e from it ffffbaf6c1e4bb20 00010000:00010000:1.0:1788572458.538119:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.538128:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572458.538169:0:18453:0:(file.c:6296:ll_getattr_dentry()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572458.538175:0:18453:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348),name=f1e.sanity-pcc 00000002:00010000:1.0:1788572458.538192:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572458.538201:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572458.538211:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572458.538217:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 1 00000080:00010000:1.0:1788572458.538222:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:1.0:1788572458.538224:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1164e 00000080:00010000:1.0:1788572458.538248:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1164e from it ffffbaf6c1e4bcb8 00010000:00010000:1.0:1788572458.538251:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572458.538258:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572458.538341:0:18453:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572458.538944:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572458.538955:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788572458.538973:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788572458.538981:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:1.0:1788572458.538986:0:18453:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572458.538990:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:1.0:1788572458.538995:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e4bbf8 00010000:00010000:1.0:1788572458.538998:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788572458.539006:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788572458.539016:0:18453:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572458.539024:0:18453:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=25 00000080:00200000:1.0:1788572458.539030:0:18453:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:1.0:1788572458.539033:0:18453:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1e.sanity-pcc, flags=265 00000080:00200000:1.0:1788572458.539038:0:18453:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 46 00000080:00200008:1.0:1788572458.539046:0:18453:0:(file.c:1090:ll_file_open()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 154402 00000002:00010000:1.0:1788572458.539059:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open flags 0104240114403 00010000:00010000:1.0:1788572458.539111:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.539119:0:18453:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572458.539130:0:18453:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572458.539144:0:18453:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da1c00 x1875453495394944/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572458.539153:0:18453:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572458.539175:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495394944:192.168.202.102@tcp:101:lfs.500 00000100:00100000:1.0:1788572458.539565:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572458.542366:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572458.542480:0:16827:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1788572458.542500:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518434304 00000100:00100000:0.0:1788572458.542511:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518434304 00000100:00100000:0.0:1788572458.542572:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 27 00000100:00100000:0.0:1788572458.542582:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a27429dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18901:x1875453518434304:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572458.542601:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572458.542671:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572458.542678:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a27429dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18901:x1875453518434304:12345-192.168.202.102@tcp:104: Request processed in 98us (323us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572458.542685:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 27 00010000:00010000:2.0:1788572458.547141:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572458.547158:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547173:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742ad400 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572458.547179:0:16838:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547188:0:16838:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547199:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547268:0:16838:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572458.547318:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547325:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572458.547360:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547368:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ad400/0x6d478a807bb1164e lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c017cd expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572458.547386:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.547388:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572458.547390:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572458.547680:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495395072:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572458.569642:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495394944:192.168.202.102@tcp:101:lfs.500 00010000:00010000:1.0:1788572458.569670:0:18453:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569687:0:18453:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 4/1,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569694:0:18453:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 4/1,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569704:0:18453:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 4/1,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569715:0:18453:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 4/1,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569738:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 5/1,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569747:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 5/1,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.569771:0:18453:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 4/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572458.569783:0:18453:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff93a258da1c00 x1875453495394944/t12884901980(12884901980) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00100000:1.0:1788572458.569796:0:18453:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da1c00 x1875453495394944/t12884901980(12884901980) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000002:00010000:1.0:1788572458.569811:0:18453:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 3/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.570199:0:18453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 8 res 8589939617/5 (0 0) 00000080:00200000:1.0:1788572458.570209:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x2000013a1:0x4:0x0] suppgids -1 500: rc 0 00000080:00200000:1.0:1788572458.570353:0:18453:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 1024 00000080:00010000:1.0:1788572458.570359:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb11655 00000080:00010000:1.0:1788572458.570366:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11655 from it ffffbaf6c1e4bbf0 00010000:00010000:1.0:1788572458.570546:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 3/0,1 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.570559:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788572458.570585:0:18453:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da1c00 x1875453495394944/t12884901980(12884901980) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:0 00000080:00200080:1.0:1788572458.570702:0:18453:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0x5:0x0](0000000000000000) cmd=c00466fc arg=7ffdf46d72cc 00800000:00000080:1.0:1788572458.573320:0:18453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000002:00000080:1.0:1788572458.573362:0:18453:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00010000:00010000:1.0:1788572458.573416:0:18453:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572458.573478:0:18453:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0x5:0x0]: losing layout lock 00010000:00010000:1.0:1788572458.573509:0:18453:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572458.573516:0:18453:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572458.573536:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495395328:192.168.202.102@tcp:103:lfs.0 00000100:00100000:1.0:1788572458.573643:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572458.574854:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495395072:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572458.577022:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495395328:192.168.202.102@tcp:103:lfs.0 00010000:00010000:1.0:1788572458.577063:0:18453:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217200/0x6d478a807bb11639 lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01780 expref: -99 pid: 18435 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:1.0:1788572458.577097:0:18453:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 58, max 7 00000100:00100000:1.0:1788572458.577105:0:18453:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da1880 x1875453495395456/t0(0) o58->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572458.577127:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495395456:192.168.202.102@tcp:58:lfs.0 00000100:00100000:1.0:1788572458.577204:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572458.582395:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495395456:192.168.202.102@tcp:58:lfs.0 00010000:00010000:1.0:1788572458.582451:0:18453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 31 res 8589939617/5 (0 0) 00000080:00010000:1.0:1788572458.582464:0:18453:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2000013a1:0x5:0x0](ffff93a251195348) 00000002:00010000:1.0:1788572458.582477:0:18453:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x2000013a1:0x5:0x0], intent: layout flags 0 00010000:00010000:1.0:1788572458.582528:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.582549:0:18453:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.582560:0:18453:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18453 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572458.582581:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495395584:192.168.202.102@tcp:101:lfs.0 00000100:00100000:1.0:1788572458.582667:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572458.584957:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495395584:192.168.202.102@tcp:101:lfs.0 00010000:00010000:1.0:1788572458.584983:0:18453:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572458.584996:0:18453:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572459.310356:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572459.310478:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518434560 00000100:00100000:1.0:1788572459.310488:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518434560 00000100:00100000:1.0:1788572459.310548:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 28 00000100:00100000:1.0:1788572459.310559:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18933:x1875453518434560:12345-192.168.202.102@tcp:107:hsm_cdtr.0 00000100:00100000:1.0:1788572459.310656:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18933:x1875453518434560:12345-192.168.202.102@tcp:107:hsm_cdtr.0 Request processed in 99us (311us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572459.310664:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 28 00000080:00200080:0.0:1788572459.311984:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c0086696 arg=7f3000000d20 00800000:00000080:0.0:1788572459.312025:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a2491b6000 uarg=0000000000000000 00000002:00000080:0.0:1788572459.312034:0:18454:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a2491b6000 uarg=0000000000000000 00000002:00000080:0.0:1788572459.312039:0:18454:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x2000013a1:0x5:0x0] from 18446744073709551615 #0 00000100:00100000:0.0:1788572459.312080:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495395712:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:0.0:1788572459.312948:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.317768:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495395712:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572459.317800:0:18454:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1e.sanity-pcc/f1e.sanity-pcc 00000080:00200080:2.0:1788572459.318123:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c01066f8 arg=6fb9e8 00800000:00000080:2.0:1788572459.318270:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c01066f8 len=16 karg=ffffbaf6c1e9bcf0 uarg=ffffbaf6c1e9bcc0 00000080:00200080:2.0:1788572459.319449:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8) cmd=c1406916 arg=7f3000000d20 00000100:00100000:2.0:1788572459.319529:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495395840:192.168.202.102@tcp:34:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.319630:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.335626:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495395840:192.168.202.102@tcp:34:lhsmtool_posix.0 00000080:00200000:2.0:1788572459.335720:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8), inode mode 4040 mask 1 00000080:00200000:2.0:1788572459.335756:0:18454:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=0x2000013a1:0x5:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110a1c8), file ffff93a243786300, open_flags 68800, mode 0 opened 0 00000080:00200000:2.0:1788572459.335764:0:18454:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=0x2000013a1:0x5:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110a1c8), intent=open 00000002:00010000:2.0:1788572459.335789:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x200000002:0x2:0x0], intent: open flags 041504001 00010000:00010000:2.0:1788572459.335855:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.335865:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572459.335876:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572459.335884:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940ad80 x1875453495396096/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572459.335893:0:18454:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572459.335915:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396096:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.336107:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.339839:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396096:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572459.339865:0:18454:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339877:0:18454:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339884:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339890:0:18454:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 3/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339913:0:18454:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339919:0:18454:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217400 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572459.339923:0:18454:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339929:0:18454:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339936:0:18454:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 5/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339942:0:18454:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572459.339949:0:18454:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 3/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.339956:0:18454:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11663 lrc: 0/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572459.339977:0:18454:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a24940ad80 x1875453495396096/t12884901981(12884901981) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572475 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572459.339991:0:18454:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940ad80 x1875453495396096/t12884901981(12884901981) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572475 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572459.340017:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572459.340056:0:18454:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x5:0x0], updating i_size 1024 00000080:00200000:2.0:1788572459.340060:0:18454:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00010000:2.0:1788572459.340065:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x0 00000080:00200008:2.0:1788572459.340086:0:18454:0:(file.c:1090:ll_file_open()) START file 0x2000013a1:0x5:0x0:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 1504000 00000002:00100000:2.0:1788572459.340111:0:18454:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940ad80 x1875453495396096/t12884901981(12884901981) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572475 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572459.340182:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 44 00000080:00200080:2.0:1788572459.340214:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c0086696 arg=7f3000000d20 00800000:00000080:2.0:1788572459.340281:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a242c48000 uarg=0000000000000000 00000002:00000080:2.0:1788572459.340691:0:18454:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a242c48000 uarg=0000000000000000 00000002:00000080:2.0:1788572459.340698:0:18454:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x2000013a1:0x5:0x0] from 18446744073709551615 #0 00000100:00100000:2.0:1788572459.340733:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396224:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.340824:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.347811:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396224:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572459.347852:0:18454:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1e.sanity-pcc/f1e.sanity-pcc 00000080:00200000:2.0:1788572459.347924:0:18454:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572459.347946:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572459.347967:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788572459.347980:0:18454:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788572459.347987:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572459.347995:0:18454:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572459.347998:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572459.348006:0:18454:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e9bbf8 00010000:00010000:2.0:1788572459.348009:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788572459.348037:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788572459.348053:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572459.348060:0:18454:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:2.0:1788572459.348066:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572459.348080:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 3 00000080:00200000:2.0:1788572459.348084:0:18454:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:5868171F:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), file ffff93a243787500, open_flags 280c2, mode 8180 opened 0 00000080:00200000:2.0:1788572459.348090:0:18454:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:5868171F:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000002:00010000:2.0:1788572459.348110:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:5868171F:fd=09,[0x2000013a1:0xa:0x0]) in obj [0x2000013a1:0x4:0x0], intent: open|creat flags 0500303 00010000:00010000:2.0:1788572459.348123:0:18454:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.348138:0:18454:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572459.348452:0:18454:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.348460:0:18454:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.348463:0:18454:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742acc00/0x6d478a807bb11647 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c017b1 expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.348488:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.348505:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572459.348515:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572459.348524:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940b800 x1875453495396480/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 680/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572459.348534:0:18454:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 3/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572459.348554:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396480:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.348686:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986d80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572459.350630:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572459.354082:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396480:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572459.354113:0:18454:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 4/0,1 mode: --/CW res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354138:0:18454:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354160:0:18454:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 4/0,1 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354173:0:18454:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354184:0:18454:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 4/0,1 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354213:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354221:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 5/1,1 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354269:0:18454:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572459.354282:0:18454:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff93a24940b800 x1875453495396480/t12884901982(12884901982) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 680/664 e 0 to 0 dl 1788572475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572459.354305:0:18454:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940b800 x1875453495396480/t12884901982(12884901982) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 680/664 e 0 to 0 dl 1788572475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572459.354319:0:18454:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572459.354332:0:18454:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354344:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/10 (0 0) 00000080:00200000:2.0:1788572459.354350:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x2000013a1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572459.354363:0:18454:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:5868171F:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=open|creat 00000080:00200000:2.0:1788572459.354393:0:18454:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000013a1:0xa:0x0](ffff93a251194b08) 00000080:00200000:2.0:1788572459.354486:0:18454:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0xa:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572459.354492:0:18454:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) 00000080:00010000:2.0:1788572459.354501:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) for lock 0x6d478a807bb1166a 00000080:00010000:2.0:1788572459.354511:0:18454:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) 00000080:00200008:2.0:1788572459.354526:0:18454:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08), flags 500302 00000002:00100000:2.0:1788572459.354545:0:18454:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940b800 x1875453495396480/t12884901982(12884901982) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572459.354576:0:18454:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1166a from it ffff93a251f0c0c0 00010000:00010000:2.0:1788572459.354581:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.354591:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572459.354616:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0xa:0x0](ffff93a251194b08), inode mode 8180 mask 40 00000080:00200000:2.0:1788572459.354647:0:18454:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572459.354661:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572459.354671:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788572459.354681:0:18454:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788572459.354688:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572459.354693:0:18454:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572459.354695:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572459.354700:0:18454:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e9bc10 00010000:00010000:2.0:1788572459.354703:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788572459.354721:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788572459.354731:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572459.354736:0:18454:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:2.0:1788572459.354740:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:2.0:1788572459.354745:0:18454:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=. :VOLATILE:0000:5868171F:fd=09, flags=0 00000080:00200000:2.0:1788572459.354749:0:18454:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 3 00000080:00200000:2.0:1788572459.354753:0:18454:0:(namei.c:2351:ll_unlink()) VFS Op:name=. :VOLATILE:0000:5868171F:fd=09, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8) 00010000:00010000:2.0:1788572459.354777:0:18454:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572459.354792:0:18454:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0xa:0x0]: losing layout lock 00010000:00010000:2.0:1788572459.354839:0:18454:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.354846:0:18454:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.354849:0:18454:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb1166a lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01828 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572459.354871:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572459.354878:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408a80 x1875453495396736/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572459.354899:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396736:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.354987:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.365475:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396736:192.168.202.102@tcp:36:lhsmtool_posix.0 00000002:00010000:2.0:1788572459.365569:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x2000013a1:0xa:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572459.365629:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 3/1,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.365640:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 3/1,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.365650:0:18454:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 3/1,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572459.365673:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396864:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.365765:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.379884:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495396864:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572459.379915:0:18454:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.379931:0:18454:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.379943:0:18454:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572459.379957:0:18454:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff93a249408a80 x1875453495396864/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 568/1072 e 0 to 0 dl 1788572475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572459.379968:0:18454:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.379982:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/10 (0 0) 00000080:00200000:2.0:1788572459.380002:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x2000013a1:0xa:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572459.380025:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) for lock 0x6d478a807bb11671 00000080:00010000:2.0:1788572459.380033:0:18454:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11671 from it ffffbaf6c1e9bc18 00010000:00010000:2.0:1788572459.380037:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.380047:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572459.380086:0:18454:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08) current size 0, valid attrs 1846, mode 93a2, uid 500, gid 500, new size 0, atime 1788572459.000000000, mtime 1788572459.000000000, ctime 1788572459.000000000 00000020:00100000:2.0:1788572459.380131:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572459.380138:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408a80 x1875453495397120/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572459.380173:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397120:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.380282:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.395945:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397120:192.168.202.102@tcp:36:lhsmtool_posix.0 00010000:00010000:2.0:1788572459.396008:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 31 res 8589939617/10 (0 0) 00000080:00010000:2.0:1788572459.396021:0:18454:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x2000013a1:0xa:0x0](ffff93a251194b08) 00000002:00010000:2.0:1788572459.396033:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0xa:0x0]) in obj [0x2000013a1:0xa:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572459.396077:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.396087:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.396096:0:18454:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572459.396126:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397248:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.396421:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.401541:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397248:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572459.401576:0:18454:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.401597:0:18454:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572459.401613:0:18454:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a249409c00 x1875453495397248/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572459.401624:0:18454:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572459.401637:0:18454:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.401649:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/10 (0 0) 00000080:00200000:2.0:1788572459.401720:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2000013a1:0xa:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572459.401738:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) for lock 0x6d478a807bb11678 00000080:00010000:2.0:1788572459.401747:0:18454:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11678 from it ffffbaf6c1e9ba68 00010000:00010000:2.0:1788572459.401751:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.401761:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.401794:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.401804:0:18454:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572459.401811:0:18454:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0xa:0x0](000000002415dad6) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572459.401916:0:18454:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0xa:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788572459.401922:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.401930:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572459.401949:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00020000:00200000:2.0:1788572459.401963:0:18454:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788572459.402074:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495397376:192.168.202.102@tcp:2:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.409728:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495397376:192.168.202.102@tcp:2:lhsmtool_posix.0 00000080:00200000:0.0:1788572459.409880:0:18454:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572459.409898:0:18454:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08) current size 0, valid attrs 10001846, mode 93a2, uid 500, gid 500, new size 0, atime 1788572459.000000000, mtime 1788572459.000000000, ctime 1788572459.000000000, rc 0 00000080:00200008:0.0:1788572459.409951:0:18454:0:(file.c:479:ll_file_release()) START file 0x2000013a1:0x5:0x0:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 1504000 00000002:00100000:0.0:1788572459.410011:0:18454:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940ad80 x1875453495396096/t12884901981(12884901981) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572475 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572459.410023:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572459.410029:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d0a80 x1875453495397632/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572459.410051:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397632:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:0.0:1788572459.412759:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d206c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572459.413356:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397632:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:0.0:1788572459.413387:0:18454:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940ad80 x1875453495396096/t12884901981(12884901981) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572475 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1788572459.413557:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=400866d6 arg=7f3000000b78 00800000:00000080:0.0:1788572459.413571:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e9bcf0 uarg=0000000000000000 00000002:00000080:0.0:1788572459.413578:0:18454:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e9bcf0 uarg=0000000000000000 00000020:00100000:0.0:1788572459.413601:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:0.0:1788572459.413621:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d0a80 x1875453495397888/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572459.413641:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397888:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:0.0:1788572459.413962:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d206c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572459.417733:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495397888:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:0.0:1788572459.417962:0:18454:0:(file.c:6296:ll_getattr_dentry()) START file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572459.417971:0:18454:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000013a1:0xa:0x0](ffff93a251194b08),name=. :VOLATILE:0000:5868171F:fd=09 00000002:00010000:0.0:1788572459.417988:0:18454:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0xa:0x0]) in obj [0x2000013a1:0xa:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572459.418001:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 30 res 8589939617/10 (0 0) 00010000:00010000:0.0:1788572459.418052:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.418063:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.418074:0:18454:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 18454 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572459.418094:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398016:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:0.0:1788572459.418351:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d206c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572459.421820:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398016:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:0.0:1788572459.421919:0:18454:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.421932:0:18454:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.421962:0:18454:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.421973:0:18454:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.421987:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.421995:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.422003:0:18454:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572459.422014:0:18454:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a2460d0a80 x1875453495398016/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572459.422025:0:18454:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572459.422038:0:18454:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.422050:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/10 (0 0) 00000080:00200000:0.0:1788572459.422057:0:18454:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0xa:0x0] suppgids -1 -1: rc 0 00000080:00200000:0.0:1788572459.422110:0:18454:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0xa:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788572459.422121:0:18454:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) for lock 0x6d478a807bb1167f 00000080:00010000:0.0:1788572459.422126:0:18454:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0xa:0x0](ffff93a251194b08) 00000080:00010000:0.0:1788572459.422130:0:18454:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1167f from it ffffbaf6c1e9bcd8 00010000:00010000:0.0:1788572459.422133:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.422142:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572459.422414:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788572459.422434:0:18454:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000013a1:0xa:0x0] 00000008:00010000:0.0:1788572459.422453:0:18454:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572459.422457:0:18454:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a250c07830, osc lock ffff93a2427cd690, flags 0x400001002 00010000:00010000:0.0:1788572459.422580:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.422592:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572459.422676:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495398144:192.168.202.102@tcp:101:lhsmtool_posix.0 00000020:00200000:0.0:1788572459.422907:0:18454:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572459.426458:0:2354:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.426569:0:2354:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.426583:0:2354:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.426599:0:2354:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572459.426613:0:2354:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572459.426624:0:2354:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788572459.426746:0:2354:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.426756:0:2354:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.426764:0:2354:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.426773:0:2354:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572459.426804:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495398144:192.168.202.102@tcp:101:lhsmtool_posix.0 00000080:00200000:0.0:1788572459.426924:0:18454:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0xa:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788572459.426930:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_release():245 00000020:00010000:0.0:1788572459.426933:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572459.426939:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250c07830 (ffff93a2516c02d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572459.426943:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba2060 (ffff93a2516c0128/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572459.426946:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250c07830 (ffff93a2516c0128/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572459.426960:0:18454:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:0.0:1788572459.426985:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=401866d8 arg=7f3007b90910 00800000:00000080:0.0:1788572459.427001:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e9bcf0 uarg=0000000000000000 00000002:00000080:0.0:1788572459.427010:0:18454:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e9bcf0 uarg=0000000000000000 00000020:00100000:0.0:1788572459.427038:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:0.0:1788572459.427047:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d0000 x1875453495398400/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572459.427071:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398400:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:0.0:1788572459.427510:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d206c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.431590:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398400:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:2.0:1788572459.431967:0:18454:0:(file.c:2650:do_file_write_iter()) START file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0], ppos: 0, count: 1024 00000080:00200000:2.0:1788572459.431984:0:18454:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788572459.431989:0:18454:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 1024 00000080:00200000:2.0:1788572459.431996:0:18454:0:(file.c:2090:ll_file_io_generic()) . :VOLATILE:0000:5868171F:fd=09: write ppos: 0, bytes: 1024 00000080:00200000:2.0:1788572459.432001:0:18454:0:(file.c:1848:ll_io_set_mirror()) . :VOLATILE:0000:5868171F:fd=09: desiginated mirror: 0 00000020:00200000:2.0:1788572459.432008:0:18454:0:(cl_io.c:214:cl_io_rw_init()) header@000000006b47e6b5[0x110, 1, [0x2000013a1:0xa:0x0] hash] 00000020:00200000:2.0:1788572459.432019:0:18454:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1024) 0 0 00000080:00200000:2.0:1788572459.432026:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572459.432041:0:18454:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788572459.432049:0:18454:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1024) 1024, 1024 00020000:00200000:2.0:1788572459.432053:0:18454:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572459.432073:0:18454:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788572459.432089:0:18454:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572459.432094:0:18454:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a258010430, osc lock ffff93a2489c7db0, flags 0x0 00010000:00010000:2.0:1788572459.432110:0:18454:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a24806f000 type 11 mode 2 res 9663677440/67 (0 4095) 00010000:00010000:2.0:1788572459.432311:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.432323:0:18454:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.432334:0:18454:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572459.432590:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398528:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.432762:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572459.436022:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572459.436111:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518434688 00000100:00100000:1.0:1788572459.436120:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518434688 00000100:00100000:1.0:1788572459.436421:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 29 00000100:00100000:1.0:1788572459.436441:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19589:x1875453518434688:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572459.436459:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572459.436602:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572459.436610:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19589:x1875453518434688:12345-192.168.202.102@tcp:104: Request processed in 171us (599us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572459.436618:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 29 00010000:00010000:2.0:1788572459.436683:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572459.436691:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.436699:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742ac200 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:2.0:1788572459.436704:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572459.436748:0:17521:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572459.436776:0:17521:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572459.436818:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.436835:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.436869:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.436878:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 0000000094d669f1/0x6d478a807bb11686 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c0186e expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.436893:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.436895:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.436897:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572459.436904:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572459.436919:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495398656:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572459.441922:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572459.441992:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518434816 00000100:00100000:0.0:1788572459.442001:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518434816 00000100:00100000:0.0:1788572459.442043:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 30 00000100:00100000:0.0:1788572459.442052:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:18885:x1875453518434816:12345-192.168.202.102@tcp:105: 00010000:00010000:0.0:1788572459.442073:0:16828:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.442086:0:16828:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.442097:0:16828:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.442108:0:16828:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.442118:0:16828:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572459.442340:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398528:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:0.0:1788572459.442544:0:16828:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000000000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.443093:0:18454:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.443114:0:18454:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572459.443126:0:18454:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x100a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.443222:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8100a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.443258:0:18454:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8100a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572459.443287:0:18454:0:(vvp_io.c:1292:vvp_io_write_start()) . :VOLATILE:0000:5868171F:fd=09: write [0, 1024) append 0 append lockpos 0 00000080:00200000:2.0:1788572459.443306:0:18454:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 1024 bytes 00000080:00200000:2.0:1788572459.443568:0:18454:0:(rw26.c:871:ll_write_end()) pos 0, len 1024, copied 1024 00000080:00200000:2.0:1788572459.443578:0:18454:0:(rw26.c:929:ll_write_end()) page@00000000daf7067c[3 00000000be4e5bcc 2 1 00000000a7dcceba 0 0 0] 00000080:00200000:2.0:1788572459.443582:0:18454:0:(rw26.c:929:ll_write_end()) vmpage @00000000216b88bd 17ffffc0001001 3:0 ffff93a27425ab60 0 lru 00000080:00200000:2.0:1788572459.443589:0:18454:0:(rw26.c:929:ll_write_end()) osc-page@00000000a97cda52 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000ebdcd6a1 000000005bf87014 > 3< 0 0 > 4< 0 0 8 5111808 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572459.443590:0:18454:0:(rw26.c:929:ll_write_end()) end page@00000000daf7067c 00000080:00200000:2.0:1788572459.443592:0:18454:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788572459.443597:0:18454:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 1024 prio 0 00000008:00010000:2.0:1788572459.443618:0:18454:0:(osc_cache.c:858:osc_extent_find()) ### extent: 000000002c259a1c ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8000a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572459.443637:0:18454:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff93a251194cb8, count 1, dirtied 1 00000080:00200000:2.0:1788572459.443642:0:18454:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 1024 bytes, tot: 1024 00000080:00200000:2.0:1788572459.443645:0:18454:0:(vvp_io.c:1358:vvp_io_write_start()) . :VOLATILE:0000:5868171F:fd=09: write bytes 0, result: 1024 00000020:00010000:2.0:1788572459.443724:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5128 (ffff93a2516c0ab8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572459.443729:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5128 (ffff93a2516c0ab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572459.443732:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a258010430 (ffff93a2516c0638/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572459.443738:0:18454:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8100a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.443759:0:18454:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 6/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8000a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572459.443775:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5128 (ffff93a2516c0ab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572459.443778:0:18454:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a258010430 (ffff93a2516c0ab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572459.443786:0:18454:0:(vvp_io.c:533:vvp_io_advance()) advancing 1024 bytes 00000080:00200000:2.0:1788572459.443792:0:18454:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788572459.443801:0:18454:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572459.443805:0:18454:0:(file.c:2238:ll_file_io_generic()) . :VOLATILE:0000:5868171F:fd=09: 2 io complete with rc: 0, result: 1024, restart: 0 00000080:00200000:2.0:1788572459.443813:0:18454:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0xa:0x0] updating i_size 1024 i_blocks 1 00000080:00200000:2.0:1788572459.443818:0:18454:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 1024 00000080:00200008:2.0:1788572459.444034:0:18454:0:(file.c:5384:ll_fsync()) START file: name=. :VOLATILE:0000:5868171F:fd=09, fid=[0x2000013a1:0xa:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/1 00000080:00200000:2.0:1788572459.444046:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572459.444055:0:18454:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788572459.444073:0:18454:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 000000002c259a1c ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8000a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572459.444085:0:18454:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000002c259a1c ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8000a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.444131:0:18454:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x8000a0000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572459.444253:0:16828:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff93a243216200) 00000008:00100000:2.0:1788572459.444679:0:18454:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff93a249409500 - object 0x240000400:67 offset 0<>1024 00000100:00100000:3.0:1788572459.444856:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495398784:192.168.202.102@tcp:4:lhsmtool_posix.0 00000100:00100000:0.0:1788572459.444928:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:18885:x1875453518434816:12345-192.168.202.102@tcp:105: Request processed in 2875us (3017us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572459.444938:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 30 00000100:00100000:3.0:1788572459.448043:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495398656:192.168.202.102@tcp:103:ldlm_bl.0 00000008:00010000:3.0:1788572459.451754:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 000000002c259a1c ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1788572459.451937:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 000000002c259a1c ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788572459.451967:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495398784:192.168.202.102@tcp:4:lhsmtool_posix.0 00000080:00200000:2.0:1788572459.452048:0:18454:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788572459.452111:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398912:192.168.202.102@tcp:44:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.452335:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.543485:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495398912:192.168.202.102@tcp:44:lhsmtool_posix.0 00000080:00200000:2.0:1788572459.543528:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572459.543542:0:18454:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788572459.543767:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495399040:192.168.202.102@tcp:16:lhsmtool_posix.0 00000100:00100000:3.0:1788572459.607365:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495399040:192.168.202.102@tcp:16:lhsmtool_posix.0 00000080:00200000:2.0:1788572459.607562:0:18454:0:(file.c:1775:ll_merge_attr_nolock()) [0x2000013a1:0xa:0x0] updating i_size 1024 i_blocks 11 00000080:00200000:2.0:1788572459.607577:0:18454:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1788572459.607598:0:18454:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x2000013a1:0xa:0x0](ffff93a251194b08) cmd=c03066b0 arg=7f3007b8fac0 00000080:00200008:2.0:1788572459.607608:0:18454:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08) current size 1024, valid attrs 100001f0, mode 0, uid 0, gid 0, new size 0, atime 1788572458.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000 00010000:00010000:2.0:1788572459.607643:0:18454:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.607679:0:18454:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.607725:0:18454:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.607732:0:18454:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.607734:0:18454:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742ac800/0x6d478a807bb1167f lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01867 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572459.607753:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572459.607762:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408380 x1875453495399296/t0(0) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572459.607783:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495399296:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.608083:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.615218:0:18454:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901984 gen 1 00000100:00100000:2.0:1788572459.615288:0:18454:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a258da1c00 x1875453495394944/t12884901980(12884901980) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000100:00100000:2.0:1788572459.615300:0:18454:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a24940b800 x1875453495396480/t12884901982(12884901982) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572475 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572459.615313:0:18454:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff93a249408380 x1875453495399296/t12884901985(12884901985) o36->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572475 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572459.615325:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495399296:192.168.202.102@tcp:36:lhsmtool_posix.0 00000080:00200000:2.0:1788572459.615375:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572459.615389:0:18454:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788572459.616683:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495399424:192.168.202.102@tcp:2:lhsmtool_posix.0 00000100:00100000:3.0:1788572459.622343:0:2356:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244a1e800: committing for last_committed 12884901894 gen 2 00000100:00100000:3.0:1788572459.622398:0:2356:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff93a249408a80 x1875453495399424/t12884901896(12884901896) o2->lustre-OST0000-osc-ffff93a244a1e800@192.168.202.102@tcp:28/4 lens 440/432 e 0 to 0 dl 1788572475 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788572459.622460:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495399424:192.168.202.102@tcp:2:lhsmtool_posix.0 00000080:00200000:2.0:1788572459.622584:0:18454:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00800000:00000080:2.0:1788572459.622602:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d4 len=392 karg=ffff93a245518600 uarg=0000000000000000 00000002:00000080:2.0:1788572459.622614:0:18454:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d4 len=392 karg=ffff93a245518600 uarg=0000000000000000 00000020:00100000:2.0:1788572459.622644:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 55, max 7 00000100:00100000:2.0:1788572459.622651:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940b100 x1875453495399680/t0(0) o55->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 472/224 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572459.622683:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495399680:192.168.202.102@tcp:55:lhsmtool_posix.0 00000100:00100000:2.0:1788572459.622769:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572459.629025:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572459.630275:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518435072 00000100:00100000:1.0:1788572459.630291:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518435072 00000100:00100000:1.0:1788572459.630361:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 31 00000100:00100000:1.0:1788572459.630371:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18901:x1875453518435072:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572459.630390:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572459.630505:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572459.630512:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18901:x1875453518435072:12345-192.168.202.102@tcp:104: Request processed in 143us (1500us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572459.630519:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 31 00010000:00010000:2.0:1788572459.631939:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572459.631952:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.631964:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216000 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572459.631987:0:16838:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.631996:0:16838:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.632006:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.632073:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.632080:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.632114:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.632122:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216000/0x6d478a807bb11671 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0xa:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c0184b expref: -99 pid: 18454 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.632143:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.632145:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.632147:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572459.632268:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572459.632291:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495399808:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572459.643893:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495399808:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572459.644573:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495399680:192.168.202.102@tcp:55:lhsmtool_posix.0 00000080:00200008:2.0:1788572459.644608:0:18454:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08) current size 1024, valid attrs 100001f0, mode 0, uid 0, gid 0, new size 0, atime 1788572458.000000000, mtime 1788572456.000000000, ctime 1788572456.000000000, rc 0 00000080:00200080:2.0:1788572459.644629:0:18454:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=400866d7 arg=7f3000000b78 00000080:00200000:2.0:1788572459.644722:0:18454:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572459.644736:0:18454:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788572459.644912:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495399936:192.168.202.102@tcp:1:lhsmtool_posix.0 00000100:00100000:1.0:1788572459.648814:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572459.648979:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518435200 00000100:00100000:0.0:1788572459.648990:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518435200 00000100:00100000:0.0:1788572459.649053:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 32 00000100:00100000:0.0:1788572459.649066:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19589:x1875453518435200:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572459.649090:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572459.649193:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572459.649200:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19589:x1875453518435200:12345-192.168.202.102@tcp:104: Request processed in 138us (398us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572459.649207:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 32 00010000:00010000:3.0:1788572459.652028:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572459.652042:0:16837:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572459.652056:0:16837:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216200 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:3.0:1788572459.652064:0:16837:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400020000 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572459.652121:0:16837:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572459.652756:0:16837:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572459.652816:0:16837:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 2/0,0 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.652896:0:16837:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572459.652967:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495400064:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572459.657572:0:16837:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 1/0,0 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.657595:0:16837:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff93a244a1e800 lock: 00000000ee602c7d/0x6d478a807bb1168d lrc: 0/0,0 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c01875 expref: -99 pid: 18454 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.657632:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.657635:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.657638:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572459.659341:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495399936:192.168.202.102@tcp:1:lhsmtool_posix.0 00800000:00000080:1.0:1788572459.659484:0:18454:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e9bcf0 uarg=0000000000000000 00000002:00000080:1.0:1788572459.659497:0:18454:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e9bcf0 uarg=0000000000000000 00000020:00100000:1.0:1788572459.659530:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:1.0:1788572459.659541:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da3b80 x1875453495400320/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572459.659569:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3b80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495400320:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:1.0:1788572459.659792:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.660571:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495400064:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572459.663979:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572459.664090:0:16827:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1788572459.664112:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518435328 00000100:00100000:0.0:1788572459.664122:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518435328 00000100:00100000:0.0:1788572459.664190:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 33 00000100:00100000:0.0:1788572459.664201:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a24913bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518435328:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572459.664219:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572459.664357:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572459.664364:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a24913bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518435328:12345-192.168.202.102@tcp:104: Request processed in 165us (396us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572459.664371:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 33 00010000:00010000:2.0:1788572459.664584:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572459.664592:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.664600:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217400 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572459.664605:0:17521:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.664614:0:17521:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.664622:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 3/0,0 mode: CR/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572459.664668:0:17521:0:(vvp_object.c:117:vvp_conf_set()) [0x2000013a1:0xa:0x0]: losing layout lock 00010000:00010000:2.0:1788572459.664713:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 2/0,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.664720:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.664742:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.664751:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217400/0x6d478a807bb11678 lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0xa:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c01859 expref: -99 pid: 18454 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572459.664772:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.664774:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.664775:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572459.664783:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572459.664796:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495400448:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572459.670542:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495400448:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572459.677088:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572459.677517:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518435712 00000100:00100000:1.0:1788572459.677527:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518435712 00000100:00100000:1.0:1788572459.677584:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 34 00000100:00100000:1.0:1788572459.677595:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518435712:12345-192.168.202.102@tcp:105: 00010000:00010000:1.0:1788572459.677612:0:16827:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 5/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.677624:0:16827:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 5/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.677633:0:16827:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 5/1,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.677642:0:16827:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 5/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.677651:0:16827:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 6/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.677691:0:16827:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 6/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x80000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572459.677722:0:16827:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff93a2742c7800) 00000100:00100000:1.0:1788572459.677782:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518435712:12345-192.168.202.102@tcp:105: Request processed in 188us (703us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572459.677790:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 34 00010000:00010000:2.0:1788572459.678089:0:18453:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.678099:0:18453:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572459.678118:0:18453:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a258da1880 x1875453495395584/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 376/344 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572459.678131:0:18453:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.678153:0:18453:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/5 (0 0) 00000080:00200000:2.0:1788572459.678160:0:18453:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x2000013a1:0x5:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572459.678179:0:18453:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb1165c 00000080:00010000:2.0:1788572459.678188:0:18453:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1165c from it ffffbaf6c1e4bc18 00010000:00010000:2.0:1788572459.678192:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572459.678202:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.678265:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.678277:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572459.678287:0:18453:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572459.678340:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495400576:192.168.202.102@tcp:49:lfs.0 00000100:00100000:2.0:1788572459.678419:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.683016:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495400576:192.168.202.102@tcp:49:lfs.0 00010000:00010000:2.0:1788572459.683116:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572459.683142:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572459.683168:0:18453:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572459.683177:0:18453:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572459.683203:0:18453:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.683214:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572459.683222:0:18453:0:(file.c:6948:ll_layout_lock_set()) ### file [0x2000013a1:0x5:0x0](00000000a292d8e5) being reconfigured ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572459.683276:0:18453:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572459.683281:0:18453:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572459.683320:0:18453:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4294967294 -> 4 00010000:00010000:2.0:1788572459.683325:0:18453:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.683332:0:18453:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00800000:00000080:2.0:1788572459.683442:0:18453:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000002:00000080:2.0:1788572459.683451:0:18453:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000020:00100000:2.0:1788572459.683551:0:18453:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 2 is allocated opc 58, max 7 00000100:00100000:2.0:1788572459.683567:0:18453:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409180 x1875453495400832/t0(0) o58->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572459.683592:0:18453:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495400832:192.168.202.102@tcp:58:lfs.0 00000100:00100000:2.0:1788572459.683684:0:18453:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572459.687220:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3b80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495400320:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:1.0:1788572459.687416:0:18454:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:5868171F:fd=09:[0x2000013a1:0xa:0x0](ffff93a251194b08), flags 500002 00000002:00100000:1.0:1788572459.687500:0:18454:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940b800 x1875453495396480/t12884901982(12884901982) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572475 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572459.687512:0:18454:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572459.687518:0:18454:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da0700 x1875453495401088/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572459.687538:0:18454:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495401088:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:1.0:1788572459.687605:0:18454:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.690031:0:18453:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18453:1875453495400832:192.168.202.102@tcp:58:lfs.0 00000080:00200008:2.0:1788572459.690098:0:18453:0:(file.c:479:ll_file_release()) START file f1e.sanity-pcc:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 154002 00010000:00010000:2.0:1788572459.690132:0:18453:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788572459.704611:0:18454:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18454:1875453495401088:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:2.0:1788572459.704656:0:18454:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940b800 x1875453495396480/t12884901982(12884901982) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572475 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572459.715485:0:18456:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572459.715513:0:18456:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572459.715531:0:18456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1788572459.715547:0:18456:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1788572459.715666:0:18456:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572459.715676:0:18456:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572459.715681:0:18456:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572459.715689:0:18456:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1e6bb98 00010000:00010000:0.0:1788572459.715694:0:18456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1788572459.715703:0:18456:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1788572459.715719:0:18456:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572459.715728:0:18456:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=24 00000080:00200000:0.0:1788572459.715733:0:18456:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 1 00000080:00200000:0.0:1788572459.715749:0:18456:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags=8 00000080:00200000:0.0:1788572459.715753:0:18456:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1e.sanity-pcc, dir=[0x2000013a1:0x4:0x0](ffff93a2511921c8), intent=getattr 00000002:00010000:0.0:1788572459.715764:0:18456:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1e.sanity-pcc,[0x0:0x0:0x0]) in obj [0x2000013a1:0x4:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572459.715802:0:18456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.715808:0:18456:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18456 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572459.715815:0:18456:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18456 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572459.715835:0:18456:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18456:1875453495401216:192.168.202.102@tcp:101:lfs.0 00000100:00100000:0.0:1788572459.715939:0:18456:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250d20540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572459.718538:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572459.718584:0:16827:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518435840 00000100:00100000:1.0:1788572459.718593:0:16827:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518435840 00000100:00100000:1.0:1788572459.718646:0:16827:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 35 00000100:00100000:1.0:1788572459.718657:0:16827:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518435840:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572459.718677:0:16827:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572459.718769:0:16827:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572459.718776:0:16827:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18902:x1875453518435840:12345-192.168.202.102@tcp:104: Request processed in 121us (653us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572459.718783:0:16827:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 35 00010000:00010000:2.0:1788572459.719461:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572459.719491:0:16838:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.719665:0:16838:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742c6800 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572459.719670:0:16838:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.719676:0:16838:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.719682:0:16838:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 3/0,0 mode: CW/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1788572459.719746:0:16838:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a258da1c00 x1875453495394944/t12884901980(12884901980) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00000020:00100000:2.0:1788572459.719756:0:16838:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572459.719761:0:16838:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940b800 x1875453495401472/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572459.719779:0:16838:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:2e29ee59-1edd-4938-bd41-1bf1d77c561a:16838:1875453495401472:192.168.202.102@tcp:35:ldlm_bl.0 00000100:00100000:2.0:1788572459.719846:0:16838:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a248986300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572459.721926:0:16838:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:2e29ee59-1edd-4938-bd41-1bf1d77c561a:16838:1875453495401472:192.168.202.102@tcp:35:ldlm_bl.0 00000002:00100000:2.0:1788572459.721951:0:16838:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a258da1c00 x1875453495394944/t12884901980(12884901980) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:0 00010000:00010000:2.0:1788572459.722013:0:16838:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 2/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.722019:0:16838:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572459.722050:0:16838:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 1/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572459.722059:0:16838:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c6800/0x6d478a807bb11655 lrc: 0/0,0 mode: --/CW res: [0x2000013a1:0x5:0x0].0x0 bits 0x5/0x5 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c017db expref: -99 pid: 18453 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788572459.722075:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495401600:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572459.722076:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.722078:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572459.722080:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572459.724769:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495401600:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572459.725422:0:18456:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18456:1875453495401216:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572459.725456:0:18456:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725488:0:18456:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725496:0:18456:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725508:0:18456:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725520:0:18456:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725530:0:18456:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725536:0:18456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725541:0:18456:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572459.725548:0:18456:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a2460d2680 x1875453495401216/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572459.725565:0:18456:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572459.725581:0:18456:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725589:0:18456:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 4 res 8589939617/5 (0 0) 00000080:00200000:3.0:1788572459.725594:0:18456:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000013a1:0x4:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572459.725649:0:18456:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:3.0:1788572459.725658:0:18456:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4 -> 4 00000080:00010000:3.0:1788572459.725663:0:18456:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb11694 00000080:00010000:3.0:1788572459.725673:0:18456:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00010000:3.0:1788572459.725676:0:18456:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11694 from it ffffbaf6c1e6bb60 00010000:00010000:3.0:1788572459.725680:0:18456:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572459.725689:0:18456:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572459.863304:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495401728:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572459.863415:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2353:1875453495401984:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572459.863490:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572459.863497:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495401856:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572459.869362:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2353:1875453495401984:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572459.869950:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2353:1875453495401856:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572459.871172:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495401728:192.168.202.102@tcp:400:kworker.0 00010000:00010000:2.0:1788572460.375359:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572460.718895:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572460.720684:0:16828:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518435968 00000100:00100000:1.0:1788572460.720703:0:16828:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518435968 00000100:00100000:1.0:1788572460.720787:0:16828:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 36 00000100:00100000:1.0:1788572460.720799:0:16828:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18933:x1875453518435968:12345-192.168.202.102@tcp:107:hsm_cdtr.0 00000100:00100000:1.0:1788572460.720915:0:16828:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:2e29ee59-1edd-4938-bd41-1bf1d77c561a+5:18933:x1875453518435968:12345-192.168.202.102@tcp:107:hsm_cdtr.0 Request processed in 119us (2029us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572460.720922:0:16828:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 36 00000080:00200080:3.0:1788572460.721710:0:18467:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=c0086696 arg=7f3000000d20 00800000:00000080:3.0:1788572460.721734:0:18467:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a2506fe000 uarg=0000000000000000 00000002:00000080:3.0:1788572460.721741:0:18467:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=c0086696 len=4128 karg=ffff93a2506fe000 uarg=0000000000000000 00000002:00000080:3.0:1788572460.721747:0:18467:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x2000013a1:0x5:0x0] from 18446744073709551615 #0 00000100:00100000:3.0:1788572460.721784:0:18467:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495402112:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:3.0:1788572460.721869:0:18467:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572460.726601:0:18467:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495402112:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:3.0:1788572460.726632:0:18467:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1e.sanity-pcc/f1e.sanity-pcc 00000080:00200000:3.0:1788572460.726779:0:18467:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8), inode mode 4040 mask 1 00000080:00200000:3.0:1788572460.726798:0:18467:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=0x2000013a1:0x5:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110a1c8), file ffff93a2454f9680, open_flags 68800, mode 0 opened 0 00000080:00200000:3.0:1788572460.726806:0:18467:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=0x2000013a1:0x5:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110a1c8), intent=open 00000002:00010000:3.0:1788572460.726833:0:18467:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x5:0x0]) in obj [0x200000002:0x2:0x0], intent: open flags 041504001 00010000:00010000:3.0:1788572460.726904:0:18467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18467 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.726915:0:18467:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18467 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572460.726927:0:18467:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572460.726936:0:18467:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064e300 x1875453495402368/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572460.726945:0:18467:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18467 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572460.726965:0:18467:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495402368:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:3.0:1788572460.727137:0:18467:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572460.733188:0:18467:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495402368:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:3.0:1788572460.733217:0:18467:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733270:0:18467:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733279:0:18467:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 4/1,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733320:0:18467:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733331:0:18467:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733354:0:18467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733362:0:18467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 5/2,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733370:0:18467:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 4/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572460.733382:0:18467:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064e300 x1875453495402368/t12884901995(12884901995) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572476 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572460.733396:0:18467:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064e300 x1875453495402368/t12884901995(12884901995) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572476 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572460.733409:0:18467:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572460.733422:0:18467:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733443:0:18467:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572460.733461:0:18467:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 2/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572460.733468:0:18467:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733475:0:18467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733492:0:18467:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572460.733513:0:18467:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1788572460.733520:0:18467:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572460.733562:0:18467:0:(namei.c:147:ll_iget()) got inode: [0x2000013a1:0x5:0x0](ffff93a251195348): rc = 0 00000080:00200000:3.0:1788572460.733574:0:18467:0:(file.c:6843:ll_layout_conf()) [0x2000013a1:0x5:0x0]: layout version change: 4 -> 4 00000080:00010000:3.0:1788572460.733579:0:18467:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) for lock 0x6d478a807bb11694 00000080:00010000:3.0:1788572460.733590:0:18467:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00200008:3.0:1788572460.733605:0:18467:0:(file.c:1090:ll_file_open()) START file 0x2000013a1:0x5:0x0:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 1504000 00000002:00100000:3.0:1788572460.733623:0:18467:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064e300 x1875453495402368/t12884901995(12884901995) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572476 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572460.733809:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572460.733817:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733825:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be8800 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572460.733829:0:17521:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733836:0:17521:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733844:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 3/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733859:0:17521:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572460.733895:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 2/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733902:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572460.733930:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733938:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a253be8800/0x6d478a807bb1169b lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c018d0 expref: -99 pid: 18467 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572460.733952:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572460.733955:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572460.733965:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572460.733981:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495402496:192.168.202.102@tcp:103:ldlm_bl.0 00000080:00010000:3.0:1788572460.734288:0:18467:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11694 from it ffff93a249131de0 00010000:00010000:3.0:1788572460.734293:0:18467:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 3/1,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572460.734313:0:18467:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572460.734343:0:18467:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348), inode mode 81a4 mask 44 00000080:00200080:3.0:1788572460.734987:0:18467:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=400866d7 arg=7f3000000b78 00800000:00000080:3.0:1788572460.734998:0:18467:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000002:00000080:3.0:1788572460.735006:0:18467:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d8 len=64 karg=ffffbaf6c1e63cf0 uarg=0000000000000000 00000020:00100000:3.0:1788572460.735028:0:18467:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:3.0:1788572460.735037:0:18467:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064f480 x1875453495402752/t0(0) o57->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572460.735055:0:18467:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495402752:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:3.0:1788572460.735103:0:18467:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167db40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572460.741849:0:18467:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495402752:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:3.0:1788572460.741910:0:18467:0:(file.c:479:ll_file_release()) START file 0x2000013a1:0x5:0x0:[0x2000013a1:0x5:0x0](ffff93a251195348), flags 1504000 00000002:00100000:3.0:1788572460.741983:0:18467:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064e300 x1875453495402368/t12884901995(12884901995) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572476 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572460.741995:0:18467:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572460.742002:0:18467:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c380 x1875453495403008/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572460.742023:0:18467:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495403008:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:3.0:1788572460.742116:0:18467:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167db40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572460.745671:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2356:1875453495402496:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572460.757727:0:18467:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18467:1875453495403008:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:3.0:1788572460.757767:0:18467:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064e300 x1875453495402368/t12884901995(12884901995) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572476 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572461.398800:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572462.428916:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572462.854266:0:18485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:2.0:1788572462.854297:0:18485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572462.854317:0:18485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788572462.854335:0:18485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788572462.854342:0:18485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572462.854366:0:18485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:2.0:1788572462.854370:0:18485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:2.0:1788572462.854378:0:18485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1ec3bf8 00010000:00010000:2.0:1788572462.854382:0:18485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788572462.854393:0:18485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:2.0:1788572462.854412:0:18485:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572462.854422:0:18485:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1e.sanity-pcc, flags=267 00000080:00200000:2.0:1788572462.854431:0:18485:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8), inode mode 41ff mask 44 00000080:00200008:2.0:1788572462.854440:0:18485:0:(file.c:1090:ll_file_open()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00000002:00010000:2.0:1788572462.854476:0:18485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000013a1:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240304001 00010000:00010000:2.0:1788572462.855178:0:18485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 18485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572462.855189:0:18485:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18485 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572462.855202:0:18485:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572462.855214:0:18485:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409180 x1875453495403264/t0(0) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572462.855223:0:18485:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 18485 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572462.861147:0:18485:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18485:1875453495403264:192.168.202.102@tcp:101:lfs.0 00000100:00100000:1.0:1788572462.861379:0:18485:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258225780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572462.871547:0:18485:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job lfs:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18485:1875453495403264:192.168.202.102@tcp:101:lfs.0 00010000:00010000:1.0:1788572462.871585:0:18485:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572462.871597:0:18485:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 4/1,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572462.871609:0:18485:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572462.871620:0:18485:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 4/1,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572462.871637:0:18485:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff93a249409180 x1875453495403264/t12884901998(12884901998) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572478 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572462.871650:0:18485:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409180 x1875453495403264/t12884901998(12884901998) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572478 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572462.871660:0:18485:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572462.871670:0:18485:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a242d92800 type 13 mode 16 res 8589939617/4 (0 0) 00000080:00200000:1.0:1788572462.871676:0:18485:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572462.871696:0:18485:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000013a1:0x4:0x0], updating i_size 16384 00000080:00010000:1.0:1788572462.871701:0:18485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000013a1:0x4:0x0](ffff93a2511921c8) for lock 0x6d478a807bb116a2 00000080:00010000:1.0:1788572462.871709:0:18485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb116a2 from it ffffbaf6c1ec3bd0 00010000:00010000:1.0:1788572462.871713:0:18485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 3/1,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572462.871745:0:18485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788572462.871775:0:18485:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409180 x1875453495403264/t12884901998(12884901998) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572478 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572462.871870:0:18485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) cmd=901066fc arg=7ffe4c200800 00000080:00200008:1.0:1788572462.871903:0:18485:0:(file.c:479:ll_file_release()) START file d1e.sanity-pcc:[0x2000013a1:0x4:0x0](ffff93a2511921c8), flags 304000 00010000:00010000:1.0:1788572462.871920:0:18485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572463.449623:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572464.477639:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572464.983659:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572464.983674:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572464.983701:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788572464.983739:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3f8c1434-73ad-4808-be79-cf461833332c:2355:1875453495403392:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.983841:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495403648:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.983869:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2355:1875453495403904:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.983895:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572464.983900:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495403520:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.983918:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495403776:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.995108:0:2355:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901998 gen 1 00000100:00100000:2.0:1788572464.995172:0:2355:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a249409180 x1875453495403264/t12884901998(12884901998) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572478 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572464.995201:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495403520:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.995216:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3f8c1434-73ad-4808-be79-cf461833332c:2355:1875453495403392:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572464.995256:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:fc253da9-16dc-43a2-a270-63fc175cf1ea:2355:1875453495403904:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572465.001171:0:2355:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244a1e800: committing for last_committed 12884901896 gen 2 00000100:00100000:2.0:1788572465.001202:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495403648:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572465.014030:0:2355:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244a1e800: committing for last_committed 12884901905 gen 1 00000100:00100000:2.0:1788572465.014066:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2355:1875453495403776:192.168.202.102@tcp:400:kworker.0 00010000:00010000:3.0:1788572465.497650:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572466.522900:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572467.545834:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572468.398427:0:18910:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572468.398445:0:18910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572468.398469:0:18910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572468.398502:0:18910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:0.0:1788572468.398582:0:18910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1788572468.398592:0:18910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572468.398601:0:18910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572468.398604:0:18910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572468.398618:0:18910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d9fcb8 00010000:00010000:0.0:1788572468.398622:0:18910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:0.0:1788572468.398632:0:18910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200008:0.0:1788572468.399890:0:18910:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572468.399898:0:18910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572468.399912:0:18910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572468.399925:0:18910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1788572468.399937:0:18910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:0.0:1788572468.399944:0:18910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572468.399949:0:18910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572468.399952:0:18910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572468.399958:0:18910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d9fcb8 00010000:00010000:0.0:1788572468.399962:0:18910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1788572468.399971:0:18910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1788572468.400127:0:18910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572468.400135:0:18910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572468.400147:0:18910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1788572468.400155:0:18910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1788572468.400162:0:18910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572468.400166:0:18910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572468.400169:0:18910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572468.400173:0:18910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d9fbb0 00010000:00010000:0.0:1788572468.400176:0:18910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1788572468.400183:0:18910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:0.0:1788572468.400191:0:18910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:0.0:1788572468.400199:0:18910:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200080:0.0:1788572468.400216:0:18910:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=8080667f arg=7ffc1025a870 00000080:00200008:0.0:1788572468.400331:0:18910:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200000:0.0:1788572468.400830:0:18910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572468.400839:0:18910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572468.400848:0:18910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:0.0:1788572468.400857:0:18910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:0.0:1788572468.400863:0:18910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572468.400872:0:18910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572468.400875:0:18910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572468.400881:0:18910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d9fbb0 00010000:00010000:0.0:1788572468.400884:0:18910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:0.0:1788572468.400892:0:18910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788572468.400903:0:18910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:0.0:1788572468.400908:0:18910:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200080:0.0:1788572468.400918:0:18910:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=8080667f arg=7ffc10259750 00000080:00200008:0.0:1788572468.400926:0:18910:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 504000 00000080:00200000:0.0:1788572468.401046:0:18910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572468.401056:0:18910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572468.401063:0:18910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788572468.401072:0:18910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788572468.401077:0:18910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572468.401082:0:18910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572468.401084:0:18910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572468.401088:0:18910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1d9fbb0 00010000:00010000:0.0:1788572468.401091:0:18910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788572468.401110:0:18910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788572468.401120:0:18910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88), inode mode 41ed mask 44 00000080:00200008:0.0:1788572468.401126:0:18910:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200008:0.0:1788572468.403038:0:18910:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200080:0.0:1788572468.478816:0:18308:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) cmd=401866d5 arg=6fb980 00800000:00000080:0.0:1788572468.478922:0:18308:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a244a1e800: cmd=401866d5 len=24 karg=ffff93a244b1ae00 uarg=0000000000000000 00000002:00000080:0.0:1788572468.478933:0:18308:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a244a1e800: cmd=401866d5 len=24 karg=ffff93a244b1ae00 uarg=0000000000000000 00000100:00100000:0.0:1788572468.478993:0:18308:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495404032:192.168.202.102@tcp:60:lhsmtool_posix.0 00000100:00100000:0.0:1788572468.479434:0:18308:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572468.490201:0:18308:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495404032:192.168.202.102@tcp:60:lhsmtool_posix.0 00000080:00200008:1.0:1788572468.492802:0:18308:0:(file.c:479:ll_file_release()) START file fid:[0x200000002:0x2:0x0](ffff93a25110a1c8), flags 100000 00000002:00100000:1.0:1788572468.492961:0:18308:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a258da2d80 x1875453495377024/t12884901920(12884901920) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572470 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572468.492974:0:18308:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572468.492980:0:18308:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da1500 x1875453495404288/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572468.493003:0:18308:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495404288:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:1.0:1788572468.493269:0:18308:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a251f8bc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572468.500584:0:18308:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:2e29ee59-1edd-4938-bd41-1bf1d77c561a:18308:1875453495404288:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:1.0:1788572468.500619:0:18308:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a258da2d80 x1875453495377024/t12884901920(12884901920) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572470 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572468.500691:0:18308:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00000080:00200008:1.0:1788572468.503889:0:18308:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), flags 100000 00010000:00010000:0.0:1788572468.568473:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572469.592276:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788572470.102534:0:37:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572470.102550:0:37:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572470.102575:0:37:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788572470.102683:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3f8c1434-73ad-4808-be79-cf461833332c:2354:1875453495404416:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572470.102773:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495404672:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572470.102801:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc253da9-16dc-43a2-a270-63fc175cf1ea:2354:1875453495404928:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572470.102826:0:2354:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572470.102832:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495404544:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572470.102855:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495404800:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572470.112288:0:2354:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901999 gen 1 00000100:00100000:0.0:1788572470.112310:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495404544:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572470.112326:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495404800:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572470.112336:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3f8c1434-73ad-4808-be79-cf461833332c:2354:1875453495404416:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572470.112347:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2354:1875453495404672:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572470.112357:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc253da9-16dc-43a2-a270-63fc175cf1ea:2354:1875453495404928:192.168.202.102@tcp:400:kworker.0 00000001:02000400:3.0:1788572470.337553:0:19059:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-pcc test 1f: Test auto RW-PCC cache with non-root user ========================================================== 21:41:08 (1788572468) 00010000:00010000:1.0:1788572470.615632:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572471.639880:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572472.666991:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572473.034969:0:19120:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572473.034986:0:19120:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:0.0:1788572473.035014:0:19120:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572473.035048:0:19120:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788572473.035063:0:19120:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788572473.035071:0:19120:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572473.035078:0:19120:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:0.0:1788572473.035082:0:19120:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:0.0:1788572473.035090:0:19120:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1dc7cb8 00010000:00010000:0.0:1788572473.035094:0:19120:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788572473.035103:0:19120:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200008:0.0:1788572473.039250:0:19120:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251395b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572473.039266:0:19120:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251395b88),name=/ 00000002:00010000:0.0:1788572473.039367:0:19120:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572473.039995:0:19120:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 19120 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.040008:0:19120:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 19120 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.040019:0:19120:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 19120 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572473.040046:0:19120:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lsof:fc253da9-16dc-43a2-a270-63fc175cf1ea:19120:1875453495405056:192.168.202.102@tcp:101:lsof.0 00000100:00100000:0.0:1788572473.040172:0:19120:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a25167d300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572473.044021:0:19120:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job lsof:fc253da9-16dc-43a2-a270-63fc175cf1ea:19120:1875453495405056:192.168.202.102@tcp:101:lsof.0 00010000:00010000:0.0:1788572473.044057:0:19120:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044069:0:19120:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044082:0:19120:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044097:0:19120:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044110:0:19120:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044124:0:19120:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044131:0:19120:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572473.044143:0:19120:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a2460d0a80 x1875453495405056/t0(0) o101->lustre-MDT0000-mdc-ffff93a244b0e800@192.168.202.102@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572489 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572473.044157:0:19120:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044189:0:19120:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a243144000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572473.044197:0:19120:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572473.044496:0:19120:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788572473.044517:0:19120:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) for lock 0x6d478a807bb116a9 00000080:00010000:0.0:1788572473.044524:0:19120:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) 00000080:00010000:0.0:1788572473.044527:0:19120:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb116a9 from it ffffbaf6c1dc7cb8 00010000:00010000:0.0:1788572473.044532:0:19120:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572473.044542:0:19120:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572473.686459:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572474.710461:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572475.094592:0:19121:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572475.094605:0:19121:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572475.094627:0:19121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572475.094648:0:19121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1788572475.094665:0:19121:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:1.0:1788572475.094685:0:19121:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572475.094694:0:19121:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572475.094698:0:19121:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:1.0:1788572475.094706:0:19121:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1dc7cb8 00010000:00010000:1.0:1788572475.094709:0:19121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1788572475.094718:0:19121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200008:1.0:1788572475.094860:0:19121:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251193a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572475.094865:0:19121:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88),name=/ 00000002:00010000:1.0:1788572475.094873:0:19121:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572475.094880:0:19121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1788572475.094890:0:19121:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:1.0:1788572475.094897:0:19121:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572475.094901:0:19121:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00010000:1.0:1788572475.094903:0:19121:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251193a88) for lock 0x6d478a807bb11567 00000080:00010000:1.0:1788572475.094907:0:19121:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11567 from it ffffbaf6c1dc7cb8 00010000:00010000:1.0:1788572475.094910:0:19121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1788572475.094928:0:19121:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:0.0:1788572475.118828:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572475.118847:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572475.118870:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572475.118876:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572475.118879:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572475.118883:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572475.118887:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572475.118890:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572475.118893:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572475.119052:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251193a88) 00000080:00200000:1.0:1788572475.119077:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000002:0x1:0x0](ffff93a251191988) 00000080:00200000:1.0:1788572475.119110:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110a1c8) 00000080:00200000:1.0:1788572475.119116:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x1:0x0](ffff93a2511700c8) 00000080:00200000:1.0:1788572475.119124:0:19121:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572475.119128:0:19121:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572475.119137:0:19121:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572475.119139:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x2:0x0](ffff93a251191148) 00000002:00100000:1.0:1788572475.119186:0:19121:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a258da0000 x1875453495365888/t12884901904(12884901904) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572450 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:100 00000020:00100000:1.0:1788572475.119196:0:19121:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572475.119200:0:19121:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da1180 x1875453495405312/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:1.0:1788572475.119218:0:19121:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495405312:192.168.202.102@tcp:35:umount.0 00000100:00100000:1.0:1788572475.120406:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24867cc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.131038:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495405312:192.168.202.102@tcp:35:umount.0 00000002:00100000:2.0:1788572475.131074:0:19121:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a258da0000 x1875453495365888/t12884901904(12884901904) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572450 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:100 00000080:00200000:2.0:1788572475.131203:0:19121:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572475.131213:0:19121:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572475.131288:0:19121:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572475.131295:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x3:0x0](ffff93a2511942c8) 00000080:00200000:2.0:1788572475.131332:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x4:0x0](ffff93a2511921c8) 00000002:00100000:2.0:1788572475.131375:0:19121:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249409180 x1875453495403264/t12884901998(12884901998) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572478 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572475.131386:0:19121:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a244a1e800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572475.131406:0:19121:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408700 x1875453495405568/t0(0) o35->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572475.131429:0:19121:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495405568:192.168.202.102@tcp:35:umount.0 00000100:00100000:2.0:1788572475.131510:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258d59180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.137940:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495405568:192.168.202.102@tcp:35:umount.0 00000002:00100000:2.0:1788572475.137999:0:19121:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249409180 x1875453495403264/t12884901998(12884901998) o101->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572478 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572475.138049:0:19121:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572475.138062:0:19121:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572475.138090:0:19121:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572475.138097:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x5:0x0](ffff93a251195348) 00000080:00200000:2.0:1788572475.138146:0:19121:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572475.138151:0:19121:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572475.138159:0:19121:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572475.138162:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x8:0x0](ffff93a251190908) 00000080:00200000:2.0:1788572475.138218:0:19121:0:(vvp_io.c:1826:vvp_io_init()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572475.138223:0:19121:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572475.138253:0:19121:0:(vvp_io.c:304:vvp_io_fini()) [0x2000013a1:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572475.138259:0:19121:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0xa:0x0](ffff93a251194b08) 00000080:00200000:2.0:1788572475.138282:0:19121:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff93a244a1e800 (sb ffff93a244a1e800) 00010000:00010000:2.0:1788572475.138322:0:19121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138332:0:19121:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138338:0:19121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138356:0:19121:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138380:0:19121:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572475.138388:0:19121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138395:0:19121:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138401:0:19121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138407:0:19121:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138429:0:19121:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1788572475.138436:0:19121:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138442:0:19121:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138448:0:19121:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.138454:0:19121:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138542:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788572475.138549:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138557:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a251f92e00 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:1.0:1788572475.138561:0:17521:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138581:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572475.138588:0:17521:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572475.138594:0:17521:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572475.138596:0:17521:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:1.0:1788572475.138629:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138636:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572475.138669:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138678:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a251f92e00/0x6d478a807bb113d8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c010fe expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138690:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572475.138693:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138699:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a251f93c00 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:1.0:1788572475.138702:0:17521:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138708:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572475.138713:0:17521:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572475.138725:0:17521:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572475.138727:0:17521:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:1.0:1788572475.138729:0:17521:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572475.138745:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138752:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572475.138758:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138764:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a251f93c00/0x6d478a807bb11402 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01105 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138773:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572475.138776:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138781:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243244e00 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:1.0:1788572475.138784:0:17521:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138791:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572475.138798:0:17521:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572475.138803:0:17521:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572475.138805:0:17521:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 00010000:00010000:1.0:1788572475.138824:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 3/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138829:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572475.138834:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572475.138841:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572475.138843:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572475.138845:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572475.138916:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495405696:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572475.139007:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495405952:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572475.139036:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572475.139042:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495405824:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572475.142473:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572475.145246:0:19121:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:2.0:1788572475.145262:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a243244e00/0x6d478a807bb114b8 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01208 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572475.145285:0:19121:0:(mgc_request.c:130:config_log_put()) dropping config log params 10000000:01000000:2.0:1788572475.145293:0:19121:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:3.0:1788572475.145294:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572475.145295:0:19121:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00000100:00100000:0.0:1788572475.145537:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495405696:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572475.145552:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495405952:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572475.145577:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495405824:192.168.202.102@tcp:103:ldlm_bl.0 00020000:01000000:2.0:1788572475.146182:0:19121:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff93a244fd5800 active: 1 00020000:01000000:2.0:1788572475.146188:0:19121:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff93a244fd3000 active: 1 00020000:01000000:2.0:1788572475.146191:0:19121:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788572475.146196:0:19121:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff93a244a1e800 00020000:01000000:2.0:1788572475.146198:0:19121:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff93a244a1e800: disconnecting target lustre-OST0001-osc-ffff93a244a1e800 00000100:00080000:2.0:1788572475.146209:0:19121:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2e29ee59-1edd-4938-bd41-1bf1d77c561a->lustre-OST0001_UUID 00000100:00080000:2.0:1788572475.146985:0:19121:0:(import.c:65:import_set_state_nolock()) ffff93a244b08800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572475.147224:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495406080:192.168.202.102@tcp:9:umount.0 00000100:00080000:3.0:1788572475.150427:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a244b08800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572475.150460:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495406080:192.168.202.102@tcp:9:umount.0 00000100:00080000:2.0:1788572475.150476:0:19121:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572475.150481:0:19121:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244a1e800: committing for last_committed 12884901905 gen 2 00000100:00100000:2.0:1788572475.150486:0:19121:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572475.150655:0:19121:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb1142c 00000020:00000080:2.0:1788572475.150674:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a244fd3000/LOV_OSC_UUID 00000020:01000000:2.0:1788572475.150728:0:19121:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff93a244a1e800 (flags='') 00000020:00000080:2.0:1788572475.150734:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572475.150751:0:19121:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff93a244a1e800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572475.150755:0:19121:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff93a2481d4d40) has no exports 00000020:00000080:0.0:1788572475.151081:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244fd3000/LOV_OSC_UUID for lustre-OST0001-osc-ffff93a244a1e800 00000008:01000000:2.0:1788572475.151155:0:19121:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff93a244a1e800: client import never connected 00000100:00080000:2.0:1788572475.151159:0:19121:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572475.151161:0:19121:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244a1e800: committing for last_committed 12884901905 gen 3 00000100:00100000:2.0:1788572475.151167:0:19121:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572475.151573:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572475.151596:0:19121:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff93a244a1e800 (uuid 2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00020000:01000000:2.0:1788572475.151601:0:19121:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff93a244a1e800 00020000:01000000:2.0:1788572475.151604:0:19121:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff93a244a1e800: disconnecting target lustre-OST0000-osc-ffff93a244a1e800 00000100:00080000:2.0:1788572475.151617:0:19121:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2e29ee59-1edd-4938-bd41-1bf1d77c561a->lustre-OST0000_UUID 00000100:00080000:2.0:1788572475.151710:0:19121:0:(import.c:65:import_set_state_nolock()) ffff93a245044800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572475.151808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495406208:192.168.202.102@tcp:9:umount.0 00000020:00000080:0.0:1788572475.152114:0:8:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a244b08800 for lustre-OST0001-osc-ffff93a244a1e800 00000020:00000080:0.0:1788572475.152123:0:8:0:(genops.c:885:class_export_put()) final put ffff93a244b08000/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:00000080:0.0:1788572475.152125:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244b08000/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-OST0001-osc-ffff93a244a1e800 00000020:01000000:0.0:1788572475.152130:0:8:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff93a244a1e800 (2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000100:00080000:3.0:1788572475.154459:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a245044800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572475.154480:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495406208:192.168.202.102@tcp:9:umount.0 00000100:00080000:2.0:1788572475.154504:0:19121:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572475.154509:0:19121:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244a1e800: committing for last_committed 12884901896 gen 3 00000100:00100000:2.0:1788572475.154638:0:19121:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572475.154839:0:19121:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11425 00000020:00000080:2.0:1788572475.154858:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a244fd5800/LOV_OSC_UUID 00000020:01000000:2.0:1788572475.154912:0:19121:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff93a244a1e800 (flags='') 00000020:00000080:2.0:1788572475.154917:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572475.154926:0:19121:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff93a244a1e800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572475.154930:0:19121:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff93a2506299c0) has no exports 00000020:00000080:0.0:1788572475.154948:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244fd5800/LOV_OSC_UUID for lustre-OST0000-osc-ffff93a244a1e800 00000008:01000000:2.0:1788572475.155168:0:19121:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff93a244a1e800: client import never connected 00000100:00080000:2.0:1788572475.155171:0:19121:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572475.155173:0:19121:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244a1e800: committing for last_committed 12884901896 gen 4 00000100:00100000:2.0:1788572475.155178:0:19121:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572475.155388:0:8:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a245044800 for lustre-OST0000-osc-ffff93a244a1e800 00000020:00000080:2.0:1788572475.155709:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572475.155717:0:19121:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff93a244a1e800 (uuid 2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000020:00000080:2.0:1788572475.155720:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a245041000/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:00000080:2.0:1788572475.155724:0:19121:0:(genops.c:844:class_export_destroy()) destroying export ffff93a245041000/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-OST0000-osc-ffff93a244a1e800 00000020:01000000:2.0:1788572475.155730:0:19121:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff93a244a1e800 (2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000020:00000080:2.0:1788572475.156011:0:19121:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb1141e 00000020:00000080:2.0:1788572475.156018:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a244fd7000/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000100:00080000:2.0:1788572475.156093:0:19121:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2e29ee59-1edd-4938-bd41-1bf1d77c561a->lustre-MDT0000_UUID 00010000:00010000:2.0:1788572475.156136:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954cc00/0x6d478a807bb114e9 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c0129b expref: -99 pid: 17470 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.156201:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954cc00/0x6d478a807bb114e9 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c0129b expref: -99 pid: 17470 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.156208:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572475.156224:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258d59a80 going to sleep for 0 seconds 00000100:00100000:2.0:1788572475.156274:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495406336:192.168.202.102@tcp:103:umount.0 00010000:00010000:2.0:1788572475.156286:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572475.156293:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572475.156298:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954cc00/0x6d478a807bb114e9 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c0129b expref: -99 pid: 17470 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.156331:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93200/0x6d478a807bb1149c lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c011b4 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.156349:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93200/0x6d478a807bb1149c lrc: 1/0,0 mode: --/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c011b4 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.156355:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572475.156360:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258d59a80 going to sleep for 0 seconds 00000100:00100000:2.0:1788572475.156365:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495406464:192.168.202.102@tcp:103:umount.0 00010000:00010000:2.0:1788572475.156370:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572475.156373:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572475.156376:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a251f93200/0x6d478a807bb1149c lrc: 0/0,0 mode: --/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c011b4 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.156390:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954c000/0x6d478a807bb11521 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x2:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c0133c expref: -99 pid: 17520 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572475.156409:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243244200/0x6d478a807bb11513 lrc: 2/0,0 mode: CW/CW res: [0x2000013a1:0x2:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01319 expref: -99 pid: 17520 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572475.156425:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954c000/0x6d478a807bb11521 lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x2:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c0133c expref: -99 pid: 17520 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572475.156431:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243244200/0x6d478a807bb11513 lrc: 1/0,0 mode: --/CW res: [0x2000013a1:0x2:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01319 expref: -99 pid: 17520 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572475.156438:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:2.0:1788572475.156443:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258d59a80 going to sleep for 0 seconds 00000100:00100000:2.0:1788572475.156448:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495406592:192.168.202.102@tcp:103:umount.0 00010000:00010000:2.0:1788572475.156453:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572475.156457:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572475.156459:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954c000/0x6d478a807bb11521 lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x2:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c0133c expref: -99 pid: 17520 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572475.156468:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243244200/0x6d478a807bb11513 lrc: 0/0,0 mode: --/CW res: [0x2000013a1:0x2:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01319 expref: -99 pid: 17520 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572475.156497:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00000080:1.0:1788572475.156584:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244fd7000/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-clilov-ffff93a244a1e800 00010000:00010000:3.0:1788572475.157502:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 1/0,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157511:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572475.157531:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a28205bc00 going to sleep for 0 seconds 00000100:00100000:3.0:1788572475.157538:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495406720:192.168.202.102@tcp:103:umount.0 00010000:00010000:3.0:1788572475.157545:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572475.157551:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788572475.157555:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954ce00/0x6d478a807bb114a3 lrc: 0/0,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c011d0 expref: -99 pid: 17447 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157577:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954de00/0x6d478a807bb11536 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x3:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01397 expref: -99 pid: 17522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.157607:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954de00/0x6d478a807bb11536 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x3:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01397 expref: -99 pid: 17522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.157614:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572475.157619:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a28205bc00 going to sleep for 0 seconds 00000100:00100000:3.0:1788572475.157624:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495406848:192.168.202.102@tcp:103:umount.0 00010000:00010000:3.0:1788572475.157629:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572475.157633:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788572475.157636:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a24954de00/0x6d478a807bb11536 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x3:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01397 expref: -99 pid: 17522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.157662:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157671:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572475.157681:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157686:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572475.157689:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:3.0:1788572475.157692:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a28205bc00 going to sleep for 0 seconds 00000100:00100000:3.0:1788572475.157695:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495406976:192.168.202.102@tcp:103:umount.0 00010000:00010000:3.0:1788572475.157698:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572475.157705:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788572475.157707:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217000/0x6d478a807bb116a2 lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x4:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c018ec expref: -99 pid: 18485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157713:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243217e00/0x6d478a807bb11575 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x4:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c014b6 expref: -99 pid: 18325 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572475.157734:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 2/0,0 mode: PR/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157740:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 2/0,0 mode: CR/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572475.157753:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 1/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157759:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 1/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572475.157764:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:3.0:1788572475.157769:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a28205bc00 going to sleep for 0 seconds 00000100:00100000:3.0:1788572475.157773:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495407104:192.168.202.102@tcp:103:umount.0 00010000:00010000:3.0:1788572475.157778:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572475.157781:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788572475.157784:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742aca00/0x6d478a807bb11694 lrc: 0/0,0 mode: --/PR res: [0x2000013a1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c018c2 expref: -99 pid: 18456 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.157794:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a2742c7800/0x6d478a807bb1165c lrc: 0/0,0 mode: --/CR res: [0x2000013a1:0x5:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c017f0 expref: -99 pid: 18453 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572475.157843:0:19121:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1788572475.157856:0:19121:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1788572475.157860:0:19121:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572475.157863:0:19121:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a28205bc00 going to sleep for 0 seconds 00000100:00100000:3.0:1788572475.157866:0:19121:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job umount:2e29ee59-1edd-4938-bd41-1bf1d77c561a:19121:1875453495407232:192.168.202.102@tcp:103:umount.0 00010000:00010000:3.0:1788572475.157869:0:19121:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572475.157872:0:19121:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788572475.157873:0:19121:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244a1e800 lock: ffff93a243216600/0x6d478a807bb11567 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01477 expref: -99 pid: 18319 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000100:00080000:3.0:1788572475.158967:0:19121:0:(import.c:65:import_set_state_nolock()) ffff93a244a1b000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572475.159172:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495407360:192.168.202.102@tcp:39:umount.0 00000100:00080000:2.0:1788572475.164953:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a244a1b000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572475.164992:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2e29ee59-1edd-4938-bd41-1bf1d77c561a:2352:1875453495407360:192.168.202.102@tcp:39:umount.0 00000100:00080000:3.0:1788572475.165030:0:19121:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:3.0:1788572475.165035:0:19121:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901999 gen 2 00000100:00100000:3.0:1788572475.165045:0:19121:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff93a25064c380 x1875453495407360/t12884902004(12884902004) o39->lustre-MDT0000-mdc-ffff93a244a1e800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572485 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572475.165069:0:19121:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572475.166999:0:19121:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11417 00000020:00000080:3.0:1788572475.167016:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a244b09000/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:00000080:3.0:1788572475.167064:0:19121:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11410 00000020:00000080:3.0:1788572475.167068:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a244b0c800/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:01000000:3.0:1788572475.167078:0:19121:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff93a244a1e800 (flags='') 00000020:00000080:3.0:1788572475.167086:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572475.167104:0:19121:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff93a244a1e800: forcing exports to disconnect: 0/1 00000020:00080000:3.0:1788572475.167108:0:19121:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff93a250628000) has no exports 00000020:00000080:3.0:1788572475.167178:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788572475.167185:0:19121:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff93a244a1e800 (uuid 2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000020:00000080:3.0:1788572475.167188:0:19121:0:(genops.c:885:class_export_put()) final put ffff93a249584800/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:00000080:3.0:1788572475.167191:0:19121:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249584800/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-clilov-ffff93a244a1e800 00000020:01000000:3.0:1788572475.167197:0:19121:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff93a244a1e800 (2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000020:01000000:3.0:1788572475.167362:0:19121:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff93a244a1e800 (flags='') 00000020:00000080:3.0:1788572475.167366:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572475.167369:0:19121:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff93a244a1e800: forcing exports to disconnect: 0/2 00000020:00080000:3.0:1788572475.167372:0:19121:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff93a25062b380) has no exports 00000020:00000080:3.0:1788572475.167414:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788572475.167417:0:19121:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff93a244a1e800 (uuid 2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000020:01000000:3.0:1788572475.167420:0:19121:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff93a244a1e800 (flags='') 00000020:00000080:3.0:1788572475.167423:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572475.167425:0:19121:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff93a244a1e800: forcing exports to disconnect: 0/3 00000020:00080000:3.0:1788572475.167427:0:19121:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff93a25062cd40) has no exports 00000020:00000080:1.0:1788572475.168615:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244b09000/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-MDT0000-mdc-ffff93a244a1e800 00000020:00000080:1.0:1788572475.168630:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244b0c800/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-clilmv-ffff93a244a1e800 00000020:00000080:1.0:1788572475.168635:0:8:0:(genops.c:885:class_export_put()) final put ffff93a249581000/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:00000080:1.0:1788572475.168638:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249581000/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-clilmv-ffff93a244a1e800 00000020:01000000:1.0:1788572475.168642:0:8:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff93a244a1e800 (2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000008:01000000:0.0:1788572475.168888:0:19121:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff93a244a1e800: client import never connected 00000100:00080000:0.0:1788572475.168893:0:19121:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572475.168897:0:19121:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244a1e800: committing for last_committed 12884901999 gen 3 00000100:00100000:0.0:1788572475.168905:0:19121:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572475.170888:0:19121:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572475.170905:0:19121:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff93a244a1e800 (uuid 2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000080:02000400:0.0:1788572475.170911:0:19121:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:00000080:1.0:1788572475.171821:0:8:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a244a1b000 for lustre-MDT0000-mdc-ffff93a244a1e800 00000020:00000080:1.0:1788572475.171833:0:8:0:(genops.c:885:class_export_put()) final put ffff93a249586800/2e29ee59-1edd-4938-bd41-1bf1d77c561a 00000020:00000080:1.0:1788572475.171837:0:8:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249586800/2e29ee59-1edd-4938-bd41-1bf1d77c561a for lustre-MDT0000-mdc-ffff93a244a1e800 00000020:01000000:1.0:1788572475.171845:0:8:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff93a244a1e800 (2e29ee59-1edd-4938-bd41-1bf1d77c561a) 00000020:01000000:0.0:1788572475.174833:0:19121:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:0.0:1788572475.174916:0:19121:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff93a244a1e800 00000020:01000004:0.0:1788572475.174921:0:19121:0:(obd_mount.c:578:lustre_stop_mgc()) MGC still has 1 references. 00000020:01000004:0.0:1788572475.174923:0:19121:0:(obd_mount.c:910:lustre_common_put_super()) MGC still in use 00000020:01000004:0.0:1788572475.174925:0:19121:0:(obd_mount.c:710:lustre_put_lsi()) put ffff93a244a1e800 1 00000020:01000004:0.0:1788572475.174928:0:19121:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff93a244a1a000 00000100:00100000:1.0:1788572475.224724:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc253da9-16dc-43a2-a270-63fc175cf1ea:2354:1875453495407488:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572475.232006:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fc253da9-16dc-43a2-a270-63fc175cf1ea:2354:1875453495407488:192.168.202.102@tcp:400:kworker.0 00000080:00200008:3.0:1788572475.521729:0:19133:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251395b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572475.521749:0:19133:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251395b88),name=/ 00000002:00010000:3.0:1788572475.521780:0:19133:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572475.521804:0:19133:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.521822:0:19133:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572475.521831:0:19133:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572475.521839:0:19133:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) 00000080:00010000:3.0:1788572475.521843:0:19133:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) for lock 0x6d478a807bb116a9 00000080:00010000:3.0:1788572475.521857:0:19133:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb116a9 from it ffffbaf6c1dc7cb8 00010000:00010000:3.0:1788572475.521861:0:19133:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.521872:0:19133:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572475.734715:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572476.758397:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572477.743960:0:19134:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251395b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572477.743976:0:19134:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251395b88),name=/ 00000002:00010000:3.0:1788572477.744000:0:19134:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572477.744020:0:19134:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572477.744035:0:19134:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572477.744043:0:19134:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572477.744051:0:19134:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) 00000080:00010000:3.0:1788572477.744055:0:19134:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) for lock 0x6d478a807bb116a9 00000080:00010000:3.0:1788572477.744062:0:19134:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb116a9 from it ffffbaf6c1dbfcb8 00010000:00010000:3.0:1788572477.744067:0:19134:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572477.744076:0:19134:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1788572477.744265:0:19134:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251395b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572477.744276:0:19134:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251395b88),name=/ 00000002:00010000:3.0:1788572477.744284:0:19134:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572477.744291:0:19134:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572477.744300:0:19134:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572477.744307:0:19134:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572477.744312:0:19134:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) 00000080:00010000:3.0:1788572477.744315:0:19134:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251395b88) for lock 0x6d478a807bb116a9 00000080:00010000:3.0:1788572477.744319:0:19134:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb116a9 from it ffffbaf6c1dbfcb8 00010000:00010000:3.0:1788572477.744323:0:19134:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572477.744330:0:19134:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572477.771789:0:19134:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251395b88) 00000080:00200000:2.0:1788572477.771826:0:19134:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000013a1:0x1:0x0](ffff93a251196c08) 00000080:00200000:2.0:1788572477.771837:0:19134:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff93a244b0e800 (sb ffff93a244b0e800) 00010000:00010000:2.0:1788572477.771880:0:19134:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.772994:0:19134:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773005:0:19134:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773012:0:19134:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773045:0:19134:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572477.773053:0:19134:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773061:0:19134:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773067:0:19134:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773073:0:19134:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773194:0:19134:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1788572477.773214:0:19134:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773221:0:19134:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773242:0:19134:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.773248:0:19134:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774383:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788572477.774398:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774420:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243245600 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:1.0:1788572477.774424:0:17521:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774433:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572477.774441:0:17521:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572477.774447:0:17521:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572477.774450:0:17521:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:1.0:1788572477.774490:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774496:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572477.774683:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774691:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a243245600/0x6d478a807bb1143a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01136 expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774709:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572477.774712:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774718:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a24954c600 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:1.0:1788572477.774720:0:17521:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774738:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572477.774744:0:17521:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572477.774749:0:17521:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572477.774751:0:17521:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:1.0:1788572477.774754:0:17521:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572477.774966:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774973:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572477.774980:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.774986:0:17521:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a24954c600/0x6d478a807bb11464 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c0113d expref: -99 pid: 16953 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.775002:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572477.775006:0:17521:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.775011:0:17521:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a251f92a00 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:1.0:1788572477.775014:0:17521:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.775020:0:17521:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572477.775037:0:17521:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572477.775044:0:17521:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572477.775046:0:17521:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 00010000:00010000:1.0:1788572477.775059:0:17521:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 3/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.775196:0:17521:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572477.775202:0:17521:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.775211:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572477.775212:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572477.775215:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572477.775425:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495407616:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572477.775674:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495407872:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572477.775707:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572477.775713:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495407744:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572477.777061:0:16837:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572477.778323:0:19134:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:2.0:1788572477.778337:0:19134:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a251f92a00/0x6d478a807bb114b1 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c01201 expref: -99 pid: 16840 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572477.778363:0:19134:0:(mgc_request.c:130:config_log_put()) dropping config log params 10000000:01000000:2.0:1788572477.778369:0:19134:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:2.0:1788572477.778371:0:19134:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:2.0:1788572477.779460:0:19134:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff93a249585800 active: 1 00020000:01000000:2.0:1788572477.779469:0:19134:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff93a249580000 active: 1 00020000:01000000:2.0:1788572477.779472:0:19134:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788572477.779478:0:19134:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff93a244b0e800 00020000:01000000:2.0:1788572477.779480:0:19134:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff93a244b0e800: disconnecting target lustre-OST0001-osc-ffff93a244b0e800 00000100:00080000:2.0:1788572477.779493:0:19134:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import fc253da9-16dc-43a2-a270-63fc175cf1ea->lustre-OST0001_UUID 00000100:00080000:2.0:1788572477.779585:0:19134:0:(import.c:65:import_set_state_nolock()) ffff93a249169000 lustre-OST0001_UUID: changing import state from IDLE to CLOSED 00000100:00080000:2.0:1788572477.779593:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572477.779597:0:19134:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244b0e800: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572477.779602:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00010000:3.0:1788572477.779838:0:16838:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000080:2.0:1788572477.780054:0:19134:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb1148e 00000020:00000080:2.0:1788572477.780082:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a249580000/LOV_OSC_UUID 00000020:01000000:2.0:1788572477.780264:0:19134:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff93a244b0e800 (flags='') 00000020:00000080:2.0:1788572477.780270:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572477.780279:0:19134:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff93a244b0e800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572477.780284:0:19134:0:(genops.c:1558:class_disconnect_exports()) OBD device 10 (ffff93a2497199c0) has no exports 00000020:00000080:3.0:1788572477.780846:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249580000/LOV_OSC_UUID for lustre-OST0001-osc-ffff93a244b0e800 00000008:01000000:2.0:1788572477.780985:0:19134:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff93a244b0e800: client import never connected 00000100:00080000:2.0:1788572477.780988:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572477.780992:0:19134:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244b0e800: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572477.780996:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572477.781288:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572477.781296:0:19134:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff93a244b0e800 (uuid fc253da9-16dc-43a2-a270-63fc175cf1ea) 00020000:01000000:2.0:1788572477.781301:0:19134:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff93a244b0e800 00020000:01000000:2.0:1788572477.781303:0:19134:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff93a244b0e800: disconnecting target lustre-OST0000-osc-ffff93a244b0e800 00000100:00080000:2.0:1788572477.781315:0:19134:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import fc253da9-16dc-43a2-a270-63fc175cf1ea->lustre-OST0000_UUID 00000100:00080000:2.0:1788572477.781403:0:19134:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-OST0000_UUID: changing import state from IDLE to CLOSED 00000100:00080000:2.0:1788572477.781408:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572477.781411:0:19134:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244b0e800: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572477.781414:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572477.781540:0:19134:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11487 00000020:00000080:2.0:1788572477.781554:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a249585800/LOV_OSC_UUID 00000020:00000080:3.0:1788572477.781736:0:36:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a249169000 for lustre-OST0001-osc-ffff93a244b0e800 00000020:00000080:3.0:1788572477.781745:0:36:0:(genops.c:885:class_export_put()) final put ffff93a24916f000/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:3.0:1788572477.781748:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a24916f000/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-OST0001-osc-ffff93a244b0e800 00000020:01000000:3.0:1788572477.781756:0:36:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff93a244b0e800 (fc253da9-16dc-43a2-a270-63fc175cf1ea) 00010000:00010000:1.0:1788572477.782370:0:17521:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:01000000:2.0:1788572477.791389:0:19134:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff93a244b0e800 (flags='') 00000020:00000080:2.0:1788572477.791400:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572477.791411:0:19134:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff93a244b0e800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572477.791416:0:19134:0:(genops.c:1558:class_disconnect_exports()) OBD device 9 (ffff93a24971cd40) has no exports 00000020:00000080:3.0:1788572477.791694:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249585800/LOV_OSC_UUID for lustre-OST0000-osc-ffff93a244b0e800 00000008:01000000:2.0:1788572477.792377:0:19134:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff93a244b0e800: client import never connected 00000100:00080000:2.0:1788572477.792384:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572477.792389:0:19134:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244b0e800: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572477.792396:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572477.793067:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572477.793096:0:19134:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff93a244b0e800 (uuid fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000020:00000080:2.0:1788572477.793102:0:19134:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11480 00000020:00000080:2.0:1788572477.793110:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a249580800/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:3.0:1788572477.794801:0:36:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a24916e000 for lustre-OST0000-osc-ffff93a244b0e800 00000020:00000080:3.0:1788572477.794819:0:36:0:(genops.c:885:class_export_put()) final put ffff93a24916f800/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:3.0:1788572477.794838:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a24916f800/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-OST0000-osc-ffff93a244b0e800 00000020:01000000:3.0:1788572477.794849:0:36:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff93a244b0e800 (fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000100:00080000:2.0:1788572477.795683:0:19134:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import fc253da9-16dc-43a2-a270-63fc175cf1ea->lustre-MDT0000_UUID 00000100:00100000:0.0:1788572477.796074:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495407744:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572477.796097:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495407616:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572477.796121:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3f8c1434-73ad-4808-be79-cf461833332c:2353:1875453495407872:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572477.796312:0:19134:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572477.796365:0:19134:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572477.796373:0:19134:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572477.796388:0:19134:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a243685c00 going to sleep for 0 seconds 00000100:00100000:2.0:1788572477.796394:0:19134:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job umount:fc253da9-16dc-43a2-a270-63fc175cf1ea:19134:1875453495408000:192.168.202.102@tcp:103:umount.0 00010000:00010000:2.0:1788572477.796402:0:19134:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572477.796407:0:19134:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572477.796411:0:19134:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244b0e800 lock: ffff93a2742ad200/0x6d478a807bb116a9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01908 expref: -99 pid: 19120 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000100:00080000:2.0:1788572477.798749:0:19134:0:(import.c:65:import_set_state_nolock()) ffff93a251f1e000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1788572477.798988:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc253da9-16dc-43a2-a270-63fc175cf1ea:2352:1875453495408128:192.168.202.102@tcp:39:umount.0 00000020:00000080:3.0:1788572477.799416:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249580800/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-clilov-ffff93a244b0e800 00000100:00080000:0.0:1788572477.807568:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a251f1e000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:0.0:1788572477.807607:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fc253da9-16dc-43a2-a270-63fc175cf1ea:2352:1875453495408128:192.168.202.102@tcp:39:umount.0 00000100:00080000:2.0:1788572477.808743:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572477.808752:0:19134:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244b0e800: committing for last_committed 12884901895 gen 2 00000100:00100000:2.0:1788572477.808778:0:19134:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff93a249408e00 x1875453495408128/t12884902005(12884902005) o39->lustre-MDT0000-mdc-ffff93a244b0e800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572487 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572477.808807:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572477.810745:0:19134:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11479 00000020:00000080:2.0:1788572477.810761:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a251f1f800/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:2.0:1788572477.810799:0:19134:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb11472 00000020:00000080:2.0:1788572477.810803:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a251f1c000/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:01000000:2.0:1788572477.810811:0:19134:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff93a244b0e800 (flags='') 00000020:00000080:2.0:1788572477.810835:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572477.810841:0:19134:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff93a244b0e800: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572477.810845:0:19134:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff93a24971b380) has no exports 00000020:00000080:2.0:1788572477.810909:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572477.810916:0:19134:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff93a244b0e800 (uuid fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000020:00000080:2.0:1788572477.810919:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a249168800/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:2.0:1788572477.810922:0:19134:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249168800/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-clilov-ffff93a244b0e800 00000020:01000000:2.0:1788572477.810928:0:19134:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff93a244b0e800 (fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000020:01000000:2.0:1788572477.811080:0:19134:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff93a244b0e800 (flags='') 00000020:00000080:2.0:1788572477.811084:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572477.811087:0:19134:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff93a244b0e800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572477.811090:0:19134:0:(genops.c:1558:class_disconnect_exports()) OBD device 7 (ffff93a2516a3380) has no exports 00000020:00000080:2.0:1788572477.811132:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572477.811136:0:19134:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff93a244b0e800 (uuid fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000020:01000000:2.0:1788572477.811139:0:19134:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff93a244b0e800 (flags='') 00000020:00000080:2.0:1788572477.811142:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572477.811144:0:19134:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff93a244b0e800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572477.811146:0:19134:0:(genops.c:1558:class_disconnect_exports()) OBD device 8 (ffff93a2516a4d40) has no exports 00000020:00000080:0.0:1788572477.812920:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a251f1f800/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-MDT0000-mdc-ffff93a244b0e800 00000020:00000080:0.0:1788572477.812939:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a251f1c000/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-clilmv-ffff93a244b0e800 00000020:00000080:0.0:1788572477.812945:0:36:0:(genops.c:885:class_export_put()) final put ffff93a251f1d000/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:0.0:1788572477.812948:0:36:0:(genops.c:844:class_export_destroy()) destroying export ffff93a251f1d000/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-clilmv-ffff93a244b0e800 00000020:01000000:0.0:1788572477.812952:0:36:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff93a244b0e800 (fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000008:01000000:2.0:1788572477.816052:0:19134:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff93a244b0e800: client import never connected 00000100:00080000:2.0:1788572477.816063:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572477.816067:0:19134:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244b0e800: committing for last_committed 12884901895 gen 3 00000100:00100000:2.0:1788572477.816076:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572477.820861:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572477.820876:0:19134:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff93a244b0e800 (uuid fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000080:02000400:2.0:1788572477.820881:0:19134:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:2.0:1788572477.820884:0:19134:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:2.0:1788572477.820943:0:19134:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff93a244b0e800 00000100:00080000:2.0:1788572477.820953:0:19134:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 3f8c1434-73ad-4808-be79-cf461833332c->MGS 00010000:00010000:2.0:1788572477.820967:0:19134:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a243244000/0x6d478a807bb113d1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0x977d9e7969c010f7 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572477.820996:0:19134:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a243244000/0x6d478a807bb113d1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0x977d9e7969c010f7 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572477.821003:0:19134:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572477.821007:0:19134:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:2.0:1788572477.821011:0:19134:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:2.0:1788572477.821053:0:19134:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a243244000/0x6d478a807bb113d1 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c010f7 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572477.821059:0:19134:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572477.821077:0:19134:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a243685c00 going to sleep for 0 seconds 00000100:00100000:2.0:1788572477.821350:0:19134:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job umount:3f8c1434-73ad-4808-be79-cf461833332c:19134:1875453495408256:192.168.202.102@tcp:103:umount.0 00010000:00010000:2.0:1788572477.821363:0:19134:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572477.821371:0:19134:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572477.821374:0:19134:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a243244000/0x6d478a807bb113d1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x977d9e7969c010f7 expref: -99 pid: 16826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572477.821403:0:19134:0:(import.c:65:import_set_state_nolock()) ffff93a249586000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572477.821462:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3f8c1434-73ad-4808-be79-cf461833332c:2352:1875453495408384:192.168.202.102@tcp:251:umount.0 00000020:00000080:3.0:1788572477.822075:0:37:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a251f1e000 for lustre-MDT0000-mdc-ffff93a244b0e800 00000020:00000080:3.0:1788572477.822092:0:37:0:(genops.c:885:class_export_put()) final put ffff93a251f1d800/fc253da9-16dc-43a2-a270-63fc175cf1ea 00000020:00000080:3.0:1788572477.822095:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff93a251f1d800/fc253da9-16dc-43a2-a270-63fc175cf1ea for lustre-MDT0000-mdc-ffff93a244b0e800 00000020:01000000:3.0:1788572477.822106:0:37:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff93a244b0e800 (fc253da9-16dc-43a2-a270-63fc175cf1ea) 00000100:00080000:0.0:1788572477.833471:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a249586000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:0.0:1788572477.833482:0:2352:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788572477.833486:0:2352:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788572477.833487:0:2352:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572477.833502:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3f8c1434-73ad-4808-be79-cf461833332c:2352:1875453495408384:192.168.202.102@tcp:251:umount.0 00000100:00080000:2.0:1788572477.838543:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572477.838553:0:19134:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572477.838557:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572477.838560:0:19134:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572477.838579:0:19134:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x6d478a807bb113ca 00000020:00000080:2.0:1788572477.838590:0:19134:0:(genops.c:885:class_export_put()) final put ffff93a249587000/3f8c1434-73ad-4808-be79-cf461833332c 00000020:01000000:2.0:1788572477.838608:0:19134:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.102@tcp (flags='') 00000020:00000080:2.0:1788572477.838615:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572477.838622:0:19134:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.102@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572477.838626:0:19134:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff93a249718000) has no exports 10000000:01000000:2.0:1788572477.838674:0:16840:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788572477.838823:0:19134:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.102@tcp: client import never connected 00000100:00080000:2.0:1788572477.838838:0:19134:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572477.838840:0:19134:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572477.838842:0:19134:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572477.838843:0:19134:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572477.842354:0:19134:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572477.842369:0:19134:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.102@tcp (uuid 3f8c1434-73ad-4808-be79-cf461833332c) 00000020:01000004:2.0:1788572477.842377:0:19134:0:(obd_mount.c:710:lustre_put_lsi()) put ffff93a244b0e800 1 00000020:01000004:2.0:1788572477.842381:0:19134:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff93a244b0a000 00000020:00000080:0.0:1788572477.843000:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff93a249587000/3f8c1434-73ad-4808-be79-cf461833332c for MGC192.168.202.102@tcp 00000020:00000080:0.0:1788572477.843017:0:37:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff93a249586000 for MGC192.168.202.102@tcp 00000020:00000080:0.0:1788572477.843026:0:37:0:(genops.c:885:class_export_put()) final put ffff93a244a1d000/3f8c1434-73ad-4808-be79-cf461833332c 00000020:00000080:0.0:1788572477.843029:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff93a244a1d000/3f8c1434-73ad-4808-be79-cf461833332c for MGC192.168.202.102@tcp 00000020:01000000:0.0:1788572477.843033:0:37:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.102@tcp (3f8c1434-73ad-4808-be79-cf461833332c) 00010000:00010000:2.0:1788572477.845225:0:16837:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:2.0:1788572477.848960:0:16838:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:2.0:1788572477.849617:0:17521:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00000100:00100000:0.0:1788572477.849830:0:16827:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff93a251615400:16827] 0 exiting: rc = 1 00000100:00100000:1.0:1788572477.850139:0:16828:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff93a2431f3e00:16828] 1 exiting: rc = 0 00000100:00100000:2.0:1788572477.850762:0:16829:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff93a2431f2000:16829] 0 exiting: rc = 1 00000100:00100000:2.0:1788572477.850822:0:16830:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff93a2431f2c00:16830] 1 exiting: rc = 0 00000100:00100000:0.0:1788572477.851129:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.851139:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572477.852549:0:16831:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff93a2516c3c00:16831] 0 exiting: rc = 1 00000100:00100000:1.0:1788572477.852660:0:16832:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff93a2516c2a00:16832] 1 exiting: rc = 0 00000100:00100000:1.0:1788572477.852715:0:16833:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff93a2489c1400:16833] 2 exiting: rc = 0 00000100:00100000:2.0:1788572477.853586:0:16834:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff93a2489c1e00:16834] 0 exiting: rc = 1 00000100:00100000:2.0:1788572477.853669:0:16835:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff93a253ba6800:16835] 1 exiting: rc = 0 00000100:00100000:2.0:1788572477.853717:0:16836:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff93a24a3cca00:16836] 2 exiting: rc = 0 00000100:00100000:0.0:1788572477.854657:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854664:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854684:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854690:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854696:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854702:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854707:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854712:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854717:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854723:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854728:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854734:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854739:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854744:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854749:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854754:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854759:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854764:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854770:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854775:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854780:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854787:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854793:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854798:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854803:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854808:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854814:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854819:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854824:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854829:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854834:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854838:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854843:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854848:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854853:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854857:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854862:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854875:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854880:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854886:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854891:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854895:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854900:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854904:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854911:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854917:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854922:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854941:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854947:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854952:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854958:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854963:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854968:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854973:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854978:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854983:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854988:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854993:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.854998:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855003:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855008:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855013:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855018:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855023:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855030:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855035:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855041:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855045:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855051:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855056:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855061:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855065:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855079:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855096:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855103:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855108:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855113:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855118:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855123:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855128:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855133:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855138:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855143:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855148:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855154:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855159:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855164:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855169:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855174:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855179:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855184:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855189:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855194:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855199:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855204:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855210:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855215:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855219:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855225:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855259:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855265:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855270:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855275:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855290:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855295:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855300:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855306:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855319:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855324:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855329:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855334:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855340:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855639:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855648:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855654:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855659:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855664:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855670:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855680:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855686:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855693:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855698:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855704:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855710:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855716:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855722:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855727:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572477.855732:0:37:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:3.0:1788572571.873711:0:20549:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:3.0:1788572571.873717:0:20549:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:3.0:1788572571.873718:0:20549:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.102@tcp:/lustre 00000020:01000004:3.0:1788572571.873719:0:20549:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:3.0:1788572571.873834:0:20549:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff93a251f1d800 00000080:01000004:3.0:1788572571.873845:0:20549:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:3.0:1788572571.873861:0:20549:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.102@tcp' 00000020:01000004:3.0:1788572571.873891:0:20549:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.102@tcp (typ=mgc) 00000020:00000080:3.0:1788572571.873904:0:20549:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.102@tcp (ffff93a2516a4d40) 00000020:00000080:3.0:1788572571.873916:0:20549:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:3.0:1788572571.874088:0:20549:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:3.0:1788572571.874109:0:20549:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:3.0:1788572571.874273:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:3.0:1788572571.874279:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:3.0:1788572571.877372:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:3.0:1788572571.877383:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:1.0:1788572571.877783:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:1.0:1788572571.877795:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572571.879820:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572571.879889:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572571.881759:0:20549:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572571.882127:0:20549:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572571.882442:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572571.882449:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:1.0:1788572571.883412:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:1.0:1788572571.883421:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:3.0:1788572571.884176:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:3.0:1788572571.884184:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:1.0:1788572571.886430:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:1.0:1788572571.886448:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:1.0:1788572571.894162:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:1.0:1788572571.894176:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:1.0:1788572571.897615:0:20549:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:1.0:1788572571.897631:0:20549:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:3.0:1788572571.901513:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572571.901517:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572571.901519:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572571.917668:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572571.917675:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572571.917678:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788572571.919060:0:20549:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a258dda800@MGC192.168.202.102@tcp: add connection 192.168.202.102@tcp at head 00010000:00010000:1.0:1788572571.919288:0:20549:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.102@tcp-0 is initialized 00000040:01000000:1.0:1788572571.919296:0:20549:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.102@tcp ctxt 1 is initialized 10000000:01000000:1.0:1788572571.922700:0:20563:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:1.0:1788572571.922777:0:20549:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.102@tcp (uuid 2f28dd17-6644-45dd-bf11-f8af88abec3c) 00000020:00000080:1.0:1788572571.922886:0:20549:0:(genops.c:1385:class_connect()) connect: client 2f28dd17-6644-45dd-bf11-f8af88abec3c, cookie 0x6d478a807bb116b7 00000100:00080000:1.0:1788572571.922904:0:20549:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:1.0:1788572571.922917:0:20549:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572571.922957:0:20549:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495408512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572571.922978:0:20549:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS 00000080:00200000:1.0:1788572571.922999:0:20549:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff93a251f1d800 (sb ffff93a251f1d800) 00000080:01000000:1.0:1788572571.923084:0:20549:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:1.0:1788572571.923086:0:20549:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:1.0:1788572571.923090:0:20549:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:1.0:1788572571.923097:0:20549:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: bf89ae46-999b-46d3-9a62-08dfe43909d0 10000000:01000000:1.0:1788572571.923573:0:20549:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:1.0:1788572571.923576:0:20549:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff93a251f1d800 10000000:01000000:1.0:1788572571.923581:0:20549:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff93a2516a4d40 10000000:01000000:1.0:1788572571.923586:0:20549:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788572571.923606:0:20549:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff93a2516a4d40 from 1 10000000:01000000:1.0:1788572571.923609:0:20549:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788572571.923640:0:20549:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572571.923651:0:20549:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572571.923661:0:20549:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572571.923682:0:20549:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff93a258da1880 x1875453495408640/t0(0) o101->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572571.923692:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2495ed600 going to sleep for 0 seconds 00010000:00010000:3.0:1788572571.923777:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572571.924029:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495408512:192.168.202.102@tcp:250:mount.lustre.0 00000100:00100000:3.0:1788572571.967312:0:2352:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.102@tcp: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572571.967331:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.102@tcp: connect to target with instance 0 00000100:00080000:3.0:1788572571.967340:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.102@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:3.0:1788572571.967346:0:2352:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:3.0:1788572571.967350:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:3.0:1788572571.967354:0:2352:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572571.967372:0:2352:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:3.0:1788572571.967384:0:2352:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000001490d4c1): req@ffff93a258da1880 x1875453495408640/t0(0) o101->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572571.967408:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495408512:192.168.202.102@tcp:250:mount.lustre.0 00000100:00100000:1.0:1788572571.969948:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2495ed600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572571.978423:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495408640:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572571.978456:0:20549:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01cf8 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572571.978468:0:20549:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01cf8 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572571.978542:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495408768:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572571.978981:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a249451180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572571.984263:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495408768:192.168.202.102@tcp:501:mount.lustre.0 10000000:01000000:2.0:1788572571.984312:0:20549:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572571.984328:0:20549:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01cf8 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572571.984339:0:20549:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a2742c7e00/0x6d478a807bb116be lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01cf8 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572571.984348:0:20549:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff93a251f1d800 10000000:01000000:2.0:1788572571.984354:0:20549:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:2.0:1788572571.984358:0:20549:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff93a251f1d800 10000000:01000000:2.0:1788572571.984361:0:20549:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572571.984365:0:20549:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff93a251f1d800 10000000:01000000:2.0:1788572571.984367:0:20549:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:2.0:1788572571.984378:0:20549:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff93a251f1d800 from 1 10000000:01000000:2.0:1788572571.984381:0:20549:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788572571.984416:0:20549:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572571.984424:0:20549:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572571.984434:0:20549:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572571.984456:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495408896:192.168.202.102@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572571.992162:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572571.992197:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495408896:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572571.992222:0:20549:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01cff expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572571.992808:0:20549:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01cff expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572571.992889:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409024:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572571.992955:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.000012:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409024:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572572.000077:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409152:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572572.000168:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.003662:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409152:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572572.004692:0:20568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20568:1875453495409280:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572572.004845:0:20568:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.008325:0:20568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20568:1875453495409280:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572572.008370:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.008383:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.008385:0:20568:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1788572572.008392:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff93a251f1d800 00000020:00000080:2.0:1788572572.008394:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572572.008407:0:20568:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff93a251f1d800 (ffff93a24971b380) 00000020:00000080:2.0:1788572572.008425:0:20568:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:2.0:1788572572.008429:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff93a251f1d800 00000020:00000080:2.0:1788572572.008431:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572572.008665:0:20568:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff93a251f1d800 (uuid bf89ae46-999b-46d3-9a62-08dfe43909d0) 00000020:01000000:2.0:1788572572.008670:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.008674:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.008676:0:20568:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1788572572.008678:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.008681:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.008683:0:20568:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572572.008686:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff93a251f1d800 00000020:00000080:2.0:1788572572.008688:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572572.008695:0:20568:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff93a251f1d800 (ffff93a249718000) 00000020:00000080:2.0:1788572572.008705:0:20568:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:2.0:1788572572.008709:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff93a251f1d800 00000020:00000080:2.0:1788572572.008711:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572572.010056:0:20568:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff93a251f1d800 (uuid bf89ae46-999b-46d3-9a62-08dfe43909d0) 00000020:01000000:2.0:1788572572.010066:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.010071:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.010073:0:20568:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572572.010082:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.010086:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.010088:0:20568:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1788572572.010091:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1788572572.010093:0:20568:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1788572572.010096:0:20568:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1788572572.010101:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.010104:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.010105:0:20568:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1788572572.010107:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.010111:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.010113:0:20568:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1788572572.010115:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572572.010121:0:20568:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.102@tcp to nid 0x20000c0a8ca66 (192.168.202.102@tcp) 00000020:01000000:2.0:1788572572.010144:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff93a251f1d800 00000020:00000080:2.0:1788572572.010146:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572572.010157:0:20568:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff93a251f1d800 (ffff93a24971cd40) 00000020:00000080:2.0:1788572572.010167:0:20568:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:2.0:1788572572.010172:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff93a251f1d800 00000020:00000080:2.0:1788572572.010174:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572572.010269:0:20568:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a24916e000@lustre-MDT0000-mdc-ffff93a251f1d800: add connection 192.168.202.102@tcp at head 00010000:00010000:2.0:1788572572.015966:0:20568:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff93a251f1d800-1 is initialized 00000040:01000000:2.0:1788572572.016210:0:20568:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff93a251f1d800 ctxt 13 is initialized 00000020:00000080:2.0:1788572572.017989:0:20568:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff93a251f1d800 (uuid bf89ae46-999b-46d3-9a62-08dfe43909d0) 00000020:01000000:2.0:1788572572.018009:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff93a251f1d800 00000020:00000080:2.0:1788572572.018014:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572572.018039:0:20568:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788572572.018074:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.018077:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.018079:0:20568:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788572572.018082:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572572.018084:0:20568:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #8 00000020:01000000:2.0:1788572572.018092:0:20568:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572572.018094:0:20568:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572572.018095:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572572.018098:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.018099:0:20568:0:(obd_config.c:1563:class_process_config()) marker 8 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572572.018102:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572572.018106:0:20568:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.102@tcp:/lustre 00000020:00000080:2.0:1788572572.018110:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.018111:0:20568:0:(obd_config.c:1563:class_process_config()) marker 13 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788572572.018115:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572572.018120:0:20568:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.102@tcp to nid 0x20000c0a8ca66 (192.168.202.102@tcp) 00000020:01000000:2.0:1788572572.018147:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff93a251f1d800 00000020:00000080:2.0:1788572572.018150:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572572.018161:0:20568:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff93a251f1d800 (ffff93a2497199c0) 00000020:00000080:2.0:1788572572.018176:0:20568:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type osc with refcount 2 00000020:01000000:2.0:1788572572.018180:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff93a251f1d800 00000020:00000080:2.0:1788572572.018183:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572572.018272:0:20568:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a24916a000@lustre-OST0000-osc-ffff93a251f1d800: add connection 192.168.202.102@tcp at head 00010000:00010000:2.0:1788572572.018941:0:20568:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff93a251f1d800-2 is initialized 00000020:00000080:2.0:1788572572.019195:0:20568:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff93a251f1d800 (uuid bf89ae46-999b-46d3-9a62-08dfe43909d0) 00000020:01000000:2.0:1788572572.019204:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff93a251f1d800 00000020:00000080:2.0:1788572572.019207:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572572.019222:0:20568:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788572572.019247:0:20568:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff93a251f1d800: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788572572.019252:0:20568:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788572572.019266:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.019269:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.019271:0:20568:0:(obd_config.c:1563:class_process_config()) marker 13 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788572572.019274:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572572.019276:0:20568:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #15 00000020:01000000:2.0:1788572572.019278:0:20568:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572572.019279:0:20568:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572572.019286:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572572.019289:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.019290:0:20568:0:(obd_config.c:1563:class_process_config()) marker 15 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572572.019292:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572572.019296:0:20568:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.102@tcp:/lustre 00000020:00000080:2.0:1788572572.019299:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.019300:0:20568:0:(obd_config.c:1563:class_process_config()) marker 19 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788572572.019303:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572572.019306:0:20568:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.102@tcp to nid 0x20000c0a8ca66 (192.168.202.102@tcp) 00000020:01000000:2.0:1788572572.019324:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff93a251f1d800 00000020:00000080:2.0:1788572572.019327:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572572.019334:0:20568:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff93a251f1d800 (ffff93a25062cd40) 00000020:00000080:2.0:1788572572.019344:0:20568:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788572572.019349:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff93a251f1d800 00000020:00000080:2.0:1788572572.019351:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572572.019406:0:20568:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a24916c800@lustre-OST0001-osc-ffff93a251f1d800: add connection 192.168.202.102@tcp at head 00010000:00010000:2.0:1788572572.019584:0:20568:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff93a251f1d800-3 is initialized 00000020:00000080:2.0:1788572572.022204:0:20568:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff93a251f1d800 (uuid bf89ae46-999b-46d3-9a62-08dfe43909d0) 00000020:01000000:2.0:1788572572.022240:0:20568:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff93a251f1d800 00000020:00000080:2.0:1788572572.022246:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572572.022274:0:20568:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1788572572.022284:0:20568:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff93a251f1d800: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1788572572.022289:0:20568:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1788572572.022299:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.022304:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.022306:0:20568:0:(obd_config.c:1563:class_process_config()) marker 19 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1788572572.022308:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.022312:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.022314:0:20568:0:(obd_config.c:1563:class_process_config()) marker 21 (0x1) lustre sys.timeout 00000020:00000080:2.0:1788572572.022317:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1788572572.022319:0:20568:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1788572572.022321:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.022331:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.022333:0:20568:0:(obd_config.c:1563:class_process_config()) marker 21 (0x2) lustre sys.timeout 00000020:01000000:2.0:1788572572.022336:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.022338:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.022340:0:20568:0:(obd_config.c:1563:class_process_config()) marker 25 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572572.022343:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572572.022397:0:20568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20568:1875453495409408:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572572.022507:0:20568:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e240 going to sleep for 11 seconds 00000100:00100000:0.0:1788572572.026313:0:20568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20568:1875453495409408:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572572.026359:0:20568:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572572.026367:0:20568:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572572.026369:0:20568:0:(obd_config.c:1563:class_process_config()) marker 25 (0x2) lustre sys.jobid_var 00000040:00080000:0.0:1788572572.026373:0:20568:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 42 count 42 00000020:01000000:3.0:1788572572.026456:0:20549:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-41 (rc=0) 10000000:01000000:3.0:1788572572.026479:0:20549:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572572.026502:0:20549:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01cff expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.026516:0:20549:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a243217000/0x6d478a807bb116c5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01cff expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572572.026536:0:20549:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff93a251f1d800 from 1 10000000:01000000:3.0:1788572572.026540:0:20549:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788572572.026580:0:20549:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.026755:0:20549:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.026766:0:20549:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572572.026791:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409536:192.168.202.102@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572572.026982:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572572.036008:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409536:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572572.036035:0:20549:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d06 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.036044:0:20549:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d06 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572572.036657:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409664:192.168.202.102@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572572.036737:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.047927:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409664:192.168.202.102@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572572.048570:0:20549:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572572.048604:0:20549:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d06 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572572.048617:0:20549:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a253be8600/0x6d478a807bb116ef lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d06 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572572.048627:0:20549:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff93a251f1d800 from 1 10000000:01000000:2.0:1788572572.048964:0:20549:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572572.049020:0:20549:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572572.049030:0:20549:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572572.049042:0:20549:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572572.049082:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409792:192.168.202.102@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572572.049213:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.054847:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409792:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572572.054881:0:20549:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572572.054891:0:20549:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572572.054965:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409920:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572572.055038:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.057820:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495409920:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572572.057887:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495410048:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572572.057970:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.060942:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20549:1875453495410048:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572572.061257:0:20572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20572:1875453495410176:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572572.061345:0:20572:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572572.063849:0:20572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20572:1875453495410176:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572572.063898:0:20572:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.063907:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.063909:0:20572:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572572.063914:0:20572:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff93a251f1d800 00000020:00000080:2.0:1788572572.063925:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572572.063932:0:20572:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.063934:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.063935:0:20572:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572572.063936:0:20572:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.063938:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.063939:0:20572:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572572.063940:0:20572:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff93a251f1d800 00000020:00000080:2.0:1788572572.063941:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572572.063992:0:20572:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572572.063995:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.063996:0:20572:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:2.0:1788572572.063998:0:20572:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572572.064000:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572572.064001:0:20572:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre mdt.lustre-MDT0000.hsm_control 00000020:01000000:2.0:1788572572.064005:0:20572:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: lustre-ffff93a251f1d800 00000020:00000080:2.0:1788572572.064007:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000100:00100000:2.0:1788572572.064060:0:20572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20572:1875453495410304:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572572.064146:0:20572:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24a38e240 going to sleep for 11 seconds 00000100:00100000:3.0:1788572572.069319:0:20572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20572:1875453495410304:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572572.069365:0:20572:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572572.069374:0:20572:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572572.069376:0:20572:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre mdt.lustre-MDT0000.hsm_control 00000040:00080000:3.0:1788572572.069381:0:20572:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:0.0:1788572572.069860:0:20549:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:0.0:1788572572.069883:0:20549:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572572.069909:0:20549:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572572.069932:0:20549:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:0.0:1788572572.069953:0:20549:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572572.069979:0:20549:0:(genops.c:1385:class_connect()) connect: client bf89ae46-999b-46d3-9a62-08dfe43909d0, cookie 0x6d478a807bb116fd 00800000:01000000:0.0:1788572572.070001:0:20549:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect bf89ae46-999b-46d3-9a62-08dfe43909d0 to lustre-clilmv-ffff93a251f1d800 00800000:01000000:0.0:1788572572.070007:0:20549:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff93a251f1d800(bf89ae46-999b-46d3-9a62-08dfe43909d0) - lustre-MDT0000_UUID, bf89ae46-999b-46d3-9a62-08dfe43909d0 00000020:00000080:0.0:1788572572.070016:0:20549:0:(genops.c:1385:class_connect()) connect: client bf89ae46-999b-46d3-9a62-08dfe43909d0, cookie 0x6d478a807bb11704 00000100:00080000:0.0:1788572572.070024:0:20549:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:0.0:1788572572.070033:0:20549:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572572.070049:0:20549:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572572.070079:0:20549:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495410432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572572.070157:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495410432:192.168.202.102@tcp:38:mount.lustre.0 00000100:00080000:0.0:1788572572.070396:0:20549:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID 00800000:01000000:0.0:1788572572.070486:0:20549:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff93a251f1d800(bf89ae46-999b-46d3-9a62-08dfe43909d0) successfully (5) 00800000:00000004:0.0:1788572572.070490:0:20549:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff93a251f1d800: age -207, max_age 0 00000100:00080000:0.0:1788572572.070533:0:2354:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff93a2460d1f80 x1875453495410560/t0(0) o41->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000080:00000004:0.0:1788572572.070570:0:20549:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff93a251f1d800: age -207, max_age 792 00800000:00000004:0.0:1788572572.070576:0:20549:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff93a251f1d800: age -207, max_age 792 00000100:00080000:0.0:1788572572.070605:0:20549:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff93a2460d1500 x1875453495410688/t0(0) o41->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572572.070614:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a242e98240 going to sleep for 0 seconds 00010000:00080000:2.0:1788572572.144656:0:2352:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff93a2460d3b80 x1875453495410432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572582 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572572.144673:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a251f1d800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572572.144683:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to target with instance 4 00000100:00080000:2.0:1788572572.144692:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff93a251f1d800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572572.144704:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1788572572.144708:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:3.0:1788572572.145027:0:2352:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000006b66ae3e): req@ffff93a2460d1f80 x1875453495410560/t0(0) o41->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572572.145067:0:2352:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000006c0cd6f1): req@ffff93a2460d1500 x1875453495410688/t0(0) o41->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572572.145091:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495410432:192.168.202.102@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572572.145211:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a242e98240 going to sleep for 11 seconds 00010000:00080000:1.0:1788572572.147097:0:2354:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff93a2460d1f80 x1875453495410560/t0(0) o41->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 1788572588 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572572.147120:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:0:1875453495410560:192.168.202.102@tcp:41:mount.lustre.0 00010000:00080000:0.0:1788572572.148070:0:20549:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff93a2460d1500 x1875453495410688/t0(0) o41->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 1788572588 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572572.148085:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:bf89ae46-999b-46d3-9a62-08dfe43909d0:20549:1875453495410688:192.168.202.102@tcp:41:mount.lustre.0 00800000:00000004:0.0:1788572572.148101:0:20549:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff93a251f1d800: update ffff93a24971d198 cache blocks 17239/17271 objects 531669/532011 00000080:00000004:0.0:1788572572.148109:0:20549:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff93a251f1d800: update ffff93a249718458 cache blocks 17239/17271 objects 531669/532011 00000100:00100000:0.0:1788572572.148151:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:bf89ae46-999b-46d3-9a62-08dfe43909d0:20549:1875453495410816:192.168.202.102@tcp:40:mount.lustre.0 00000100:00100000:0.0:1788572572.148224:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a243952f00 going to sleep for 11 seconds 00010000:00080000:0.0:1788572572.153572:0:20549:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff93a2460d1500 x1875453495410816/t0(0) o40->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 440/448 e 0 to 0 dl 1788572588 ref 2 fl Rpc:RQU/209/0 rc 0/0 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572572.153607:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:bf89ae46-999b-46d3-9a62-08dfe43909d0:20549:1875453495410816:192.168.202.102@tcp:40:mount.lustre.0 00000080:00000004:0.0:1788572572.153634:0:20549:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:0.0:1788572572.153652:0:20549:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 3407872 00020000:01000000:0.0:1788572572.153655:0:20549:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:0.0:1788572572.153670:0:20549:0:(genops.c:1385:class_connect()) connect: client bf89ae46-999b-46d3-9a62-08dfe43909d0, cookie 0x6d478a807bb1170b 00000020:00000080:0.0:1788572572.153690:0:20549:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x6d478a807bb11712 00000100:00080000:0.0:1788572572.153698:0:20549:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:0.0:1788572572.153707:0:20549:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572572.153718:0:20549:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572572.153742:0:20549:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495410944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572572.153770:0:20549:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID 00020000:01000000:0.0:1788572572.153787:0:20549:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff93a251f1d800) active 00000020:00000080:0.0:1788572572.153816:0:20549:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x6d478a807bb11719 00000100:00080000:0.0:1788572572.153821:0:20549:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:0.0:1788572572.153841:0:20549:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff93a251f1d800: import ffff93a24916c800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572572.153847:0:20549:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572572.153860:0:20549:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495411072/t0(0) o8->lustre-OST0001-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572572.153869:0:20549:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0001_UUID 00020000:01000000:0.0:1788572572.153875:0:20549:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff93a251f1d800) active 00000100:00100000:0.0:1788572572.153915:0:20549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:bf89ae46-999b-46d3-9a62-08dfe43909d0:20549:1875453495411200:192.168.202.102@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572572.154024:0:20549:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a243952f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572572.154099:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495410944:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572572.154137:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495411072:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572572.157704:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a251f1d800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572572.157733:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff93a251f1d800: connect to target with instance 4 00000100:00080000:3.0:1788572572.157744:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff93a251f1d800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572572.157757:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:3.0:1788572572.157761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572572.157768:0:2352:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:3.0:1788572572.157773:0:2352:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572572.157777:0:2352:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572572.157792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495410944:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572572.157807:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a251f1d800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572572.157822:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff93a251f1d800: connect to target with instance 4 00000100:00080000:3.0:1788572572.157826:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff93a251f1d800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572572.157831:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:3.0:1788572572.157833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916c800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572572.157836:0:2352:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:3.0:1788572572.157838:0:2352:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572572.157841:0:2352:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572572.157846:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495411072:192.168.202.102@tcp:8:mount.lustre.0 00010000:00080000:0.0:1788572572.210088:0:20549:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff93a2460d1180 x1875453495411200/t0(0) o33->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 448/464 e 0 to 0 dl 1788572588 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'mount.lustre.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572572.210108:0:20549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:bf89ae46-999b-46d3-9a62-08dfe43909d0:20549:1875453495411200:192.168.202.102@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788572572.210326:0:20549:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00200000:0.0:1788572572.210335:0:20549:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:02000400:0.0:1788572572.210922:0:20549:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_4_g27fe0b7 00000080:00000004:0.0:1788572572.227145:0:20549:0:(super25.c:182:lustre_fill_super()) 192.168.202.102@tcp:/lustre: Mount complete 00000080:00200008:2.0:1788572572.494759:0:20593:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572572.494770:0:20593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572572.494799:0:20593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572572.494858:0:20593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 20593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572572.494868:0:20593:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 20593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572572.494877:0:20593:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 20593 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572572.494902:0:20593:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job mkdir:bf89ae46-999b-46d3-9a62-08dfe43909d0:20593:1875453495411328:192.168.202.102@tcp:101:mkdir.0 00000100:00100000:2.0:1788572572.494978:0:20593:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a249451b40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572572.502590:0:20593:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job mkdir:bf89ae46-999b-46d3-9a62-08dfe43909d0:20593:1875453495411328:192.168.202.102@tcp:101:mkdir.0 00010000:00010000:3.0:1788572572.502706:0:20593:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502719:0:20593:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502731:0:20593:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502743:0:20593:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502757:0:20593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502765:0:20593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502772:0:20593:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572572.502785:0:20593:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a24940b800 x1875453495411328/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572588 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572572.502799:0:20593:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502831:0:20593:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572572.502837:0:20593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572572.502866:0:20593:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:3.0:1788572572.502871:0:20593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572572.502877:0:20593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572572.502880:0:20593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c36bbcb8 00010000:00010000:3.0:1788572572.502883:0:20593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.502892:0:20593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572572.950426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572573.977891:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572574.870809:0:20645:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572574.870823:0:20645:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572574.870842:0:20645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572574.870866:0:20645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572574.870881:0:20645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572574.870890:0:20645:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572574.870906:0:20645:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572574.870909:0:20645:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:0.0:1788572574.870917:0:20645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c0d53cb8 00010000:00010000:0.0:1788572574.870922:0:20645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572574.870931:0:20645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200008:0.0:1788572574.872023:0:20645:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572574.872032:0:20645:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572574.872045:0:20645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572574.872058:0:20645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572574.872069:0:20645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:0.0:1788572574.872076:0:20645:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572574.872083:0:20645:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572574.872085:0:20645:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:0.0:1788572574.872090:0:20645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c0d53cb8 00010000:00010000:0.0:1788572574.872094:0:20645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572574.872102:0:20645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572574.872225:0:20645:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572574.873660:0:20645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572574.873676:0:20645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572574.873690:0:20645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572574.873697:0:20645:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572574.874983:0:20645:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572574.874992:0:20645:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:0.0:1788572574.875002:0:20645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c0d53bb0 00010000:00010000:0.0:1788572574.875008:0:20645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572574.875019:0:20645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572574.875034:0:20645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 44 00000080:00200008:0.0:1788572574.875046:0:20645:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 100000 00000080:00200080:0.0:1788572574.875177:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:0.0:1788572574.875194:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a249138c00 uarg=0000000000000000 00800000:00000004:0.0:1788572574.875201:0:20645:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff93a251f1d800: age 793, max_age 795 00000100:00100000:0.0:1788572574.877760:0:20645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:20645:1875453495411456:192.168.202.102@tcp:41:lfs.0 00000100:00100000:0.0:1788572574.878111:0:20645:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2487f2c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572574.888131:0:20645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:20645:1875453495411456:192.168.202.102@tcp:41:lfs.0 00800000:00000004:1.0:1788572574.888160:0:20645:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff93a251f1d800: update ffff93a24971d198 cache blocks 17239/17271 objects 531669/532011 00000080:00200080:1.0:1788572574.888281:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888303:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888310:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888326:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888331:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888336:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888340:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888344:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888347:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888351:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888355:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888358:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888362:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888365:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888368:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888371:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888375:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888388:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888393:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888397:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888400:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888404:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888408:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888411:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888415:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888419:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888422:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888426:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888436:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888440:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888444:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888447:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888452:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888455:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888460:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888463:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888467:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888471:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888489:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888494:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888498:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888502:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888505:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888509:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888513:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888516:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888520:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888524:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888528:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888531:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888535:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888548:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888961:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888975:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888980:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888984:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888989:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.888992:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.888996:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889000:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889003:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889007:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889011:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889014:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889018:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889021:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889025:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889028:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889031:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889035:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889039:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889042:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889046:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889049:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889053:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889056:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889060:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889064:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889073:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889077:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889081:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889085:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889089:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889092:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889097:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889100:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889104:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889108:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889112:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889115:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889119:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889122:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889127:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889130:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889134:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889137:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889141:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889144:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889147:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889151:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889154:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889157:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889161:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889171:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889176:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889180:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889185:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889189:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889193:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889196:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889199:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889203:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889206:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889209:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889212:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889215:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889219:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889222:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889225:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889307:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889313:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889316:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889319:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889322:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889325:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889328:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889331:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00800000:00000080:1.0:1788572574.889334:0:20645:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.889345:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.889351:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00020000:00000004:1.0:1788572574.889359:0:20645:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff93a251f1d800: age -207, max_age 795 00000100:00100000:1.0:1788572574.889405:0:20645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:20645:1875453495411584:192.168.202.102@tcp:13:lfs.0 00000100:00100000:1.0:1788572574.889867:0:20645:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2495ed240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572574.894970:0:20645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:20645:1875453495411584:192.168.202.102@tcp:13:lfs.0 00020000:00000004:1.0:1788572574.895006:0:20645:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff93a251f1d800: update ffff93a249719e18 cache blocks 3679/3683 objects 117792/118188 00000080:00200080:1.0:1788572574.895137:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.895155:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00020000:00000004:1.0:1788572574.895163:0:20645:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff93a251f1d800: age -207, max_age 795 00000100:00100000:1.0:1788572574.895203:0:20645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:20645:1875453495411712:192.168.202.102@tcp:13:lfs.0 00000100:00100000:1.0:1788572574.895295:0:20645:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2495ed240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572574.900373:0:20645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:20645:1875453495411712:192.168.202.102@tcp:13:lfs.0 00020000:00000004:1.0:1788572574.900533:0:20645:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff93a251f1d800: update ffff93a25062d198 cache blocks 3671/3683 objects 117536/117931 00000080:00200080:1.0:1788572574.901007:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901024:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901032:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901036:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901041:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901044:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901048:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901052:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901056:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901059:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901078:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901082:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901087:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901090:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901094:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901098:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901102:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901105:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901108:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901112:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901115:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901118:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901122:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901125:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901130:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901133:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901137:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901140:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901144:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901147:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901150:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901154:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901158:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901161:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901165:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901175:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901181:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901185:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901188:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901192:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901196:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901199:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901203:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901206:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901210:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901213:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901216:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.901219:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.901224:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903395:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903426:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903440:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903446:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903450:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903454:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903457:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903461:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903464:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903466:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903470:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903503:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903508:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903512:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903515:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903519:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903522:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903526:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903530:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903533:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903545:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903556:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903561:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903565:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903568:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903572:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903575:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903578:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903581:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903584:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903586:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903590:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903593:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903597:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903600:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903604:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903613:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903619:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903622:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903626:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903629:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903633:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903636:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903640:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903643:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903646:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903650:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903653:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903657:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903661:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903664:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903668:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903672:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903676:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903679:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903683:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903685:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903688:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903692:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903695:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903698:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903720:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903725:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903730:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903734:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903737:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903740:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903744:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903747:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903750:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903753:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903757:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903760:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903764:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903767:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200080:1.0:1788572574.903771:0:20645:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c00866a4 arg=7ffeb25e9b90 00020000:00000080:1.0:1788572574.903774:0:20645:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff93a251f1d800: cmd=c00866a4 len=592 karg=ffff93a250b8ec00 uarg=0000000000000000 00000080:00200008:1.0:1788572574.903790:0:20645:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 100000 00010000:00010000:1.0:1788572575.000788:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:01000004:0.0:1788572575.178415:0:20679:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:0.0:1788572575.178425:0:20679:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:0.0:1788572575.178426:0:20679:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.102@tcp:/lustre 00000020:01000004:0.0:1788572575.178428:0:20679:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:0.0:1788572575.181141:0:20679:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff93a244fd4000 00000080:01000004:0.0:1788572575.181162:0:20679:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:0.0:1788572575.181189:0:20679:0:(obd_mount.c:403:lustre_start_mgc()) MGC192.168.202.102@tcp:Set MGC reconnect 1 10000000:01000000:0.0:1788572575.181193:0:20679:0:(mgc_request.c:1071:mgc_set_info_async()) InitRecov MGC192.168.202.102@tcp 1/d0:i0:r0:or0:FULL 00000080:00200000:0.0:1788572575.181200:0:20679:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff93a244fd4000 (sb ffff93a244fd4000) 00000080:01000000:0.0:1788572575.181296:0:20679:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:0.0:1788572575.181298:0:20679:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788572575.181303:0:20679:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:0.0:1788572575.181310:0:20679:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 96693705-5e29-4404-974e-e904d01a815e 10000000:01000000:0.0:1788572575.185814:0:20679:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788572575.185820:0:20679:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff93a244fd4000 10000000:01000000:0.0:1788572575.185830:0:20679:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff93a244fd4000 10000000:01000000:0.0:1788572575.185834:0:20679:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:0.0:1788572575.185838:0:20679:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff93a244fd4000 10000000:01000000:0.0:1788572575.185841:0:20679:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572575.185844:0:20679:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff93a244fd4000 10000000:01000000:0.0:1788572575.185846:0:20679:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:0.0:1788572575.185849:0:20679:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff93a244fd4000 from 1 10000000:01000000:0.0:1788572575.185852:0:20679:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788572575.185900:0:20679:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.185913:0:20679:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.185926:0:20679:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572575.185954:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495411840:192.168.202.102@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572575.186214:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.211079:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495411840:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572575.211124:0:20679:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d37 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.211138:0:20679:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d37 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572575.211223:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495411968:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572575.211336:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572575.223418:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495411968:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572575.223512:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412096:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572575.223801:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbba80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.237277:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412096:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572575.238135:0:20684:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20684:1875453495412224:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572575.238266:0:20684:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.251225:0:20684:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20684:1875453495412224:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572575.255450:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572575.255751:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.255755:0:20684:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:3.0:1788572575.255767:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff93a244fd4000 00000020:00000080:3.0:1788572575.255770:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572575.255785:0:20684:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff93a244fd4000 (ffff93a2516a19c0) 00000020:00000080:3.0:1788572575.255807:0:20684:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type lov with refcount 2 00000020:01000000:3.0:1788572575.255813:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff93a244fd4000 00000020:00000080:3.0:1788572575.255816:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788572575.256169:0:20684:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff93a244fd4000 (uuid 96693705-5e29-4404-974e-e904d01a815e) 00000020:01000000:3.0:1788572575.256175:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572575.256179:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.256181:0:20684:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:3.0:1788572575.256184:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572575.256187:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.256188:0:20684:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:3.0:1788572575.256192:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff93a244fd4000 00000020:00000080:3.0:1788572575.256195:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572575.256202:0:20684:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff93a244fd4000 (ffff93a2516a0000) 00000020:00000080:3.0:1788572575.256211:0:20684:0:(obd_config.c:720:class_attach_name()) OBD: dev 7 attached type lmv with refcount 2 00000020:01000000:3.0:1788572575.256216:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff93a244fd4000 00000020:00000080:3.0:1788572575.256218:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572575.265260:0:20684:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff93a244fd4000 (uuid 96693705-5e29-4404-974e-e904d01a815e) 00000020:01000000:0.0:1788572575.265278:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572575.265285:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.265287:0:20684:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572575.265292:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572575.265296:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.265298:0:20684:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:0.0:1788572575.265301:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:0.0:1788572575.265303:0:20684:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:0.0:1788572575.265306:0:20684:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:0.0:1788572575.265311:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572575.265313:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.265315:0:20684:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:0.0:1788572575.265317:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572575.265320:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.265322:0:20684:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:0.0:1788572575.265324:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572575.265330:0:20684:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.102@tcp to nid 0x20000c0a8ca66 (192.168.202.102@tcp) 00000020:01000000:0.0:1788572575.265370:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff93a244fd4000 00000020:00000080:0.0:1788572575.265372:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572575.265385:0:20684:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff93a244fd4000 (ffff93a244b7cd40) 00000020:00000080:0.0:1788572575.265405:0:20684:0:(obd_config.c:720:class_attach_name()) OBD: dev 8 attached type mdc with refcount 2 00000020:01000000:0.0:1788572575.265409:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff93a244fd4000 00000020:00000080:0.0:1788572575.265411:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572575.267821:0:20684:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a24a13b000@lustre-MDT0000-mdc-ffff93a244fd4000: add connection 192.168.202.102@tcp at head 00010000:00010000:0.0:1788572575.277260:0:20684:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff93a244fd4000-4 is initialized 00000040:01000000:0.0:1788572575.280248:0:20684:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff93a244fd4000 ctxt 13 is initialized 00000020:00000080:0.0:1788572575.280322:0:20684:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff93a244fd4000 (uuid 96693705-5e29-4404-974e-e904d01a815e) 00000020:01000000:0.0:1788572575.280342:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff93a244fd4000 00000020:00000080:0.0:1788572575.280347:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572575.280382:0:20684:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:0.0:1788572575.280429:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572575.280433:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.280435:0:20684:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:0.0:1788572575.280438:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572575.280441:0:20684:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #8 00000020:01000000:0.0:1788572575.280443:0:20684:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572575.280444:0:20684:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572575.280446:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572575.280449:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.280450:0:20684:0:(obd_config.c:1563:class_process_config()) marker 8 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572575.280452:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572575.280471:0:20684:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.102@tcp:/lustre 00000020:00000080:0.0:1788572575.280475:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572575.280477:0:20684:0:(obd_config.c:1563:class_process_config()) marker 13 (0x1) lustre-OST0000 add osc 00000020:00000080:0.0:1788572575.280480:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572575.280485:0:20684:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.102@tcp to nid 0x20000c0a8ca66 (192.168.202.102@tcp) 00000020:01000000:0.0:1788572575.280527:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff93a244fd4000 00000020:00000080:0.0:1788572575.280530:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572575.281683:0:20684:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff93a244fd4000 (ffff93a244b799c0) 00000020:00000080:0.0:1788572575.281820:0:20684:0:(obd_config.c:720:class_attach_name()) OBD: dev 9 attached type osc with refcount 2 00000020:01000000:0.0:1788572575.281835:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff93a244fd4000 00000020:00000080:0.0:1788572575.281839:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572575.281930:0:20684:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a24a13b800@lustre-OST0000-osc-ffff93a244fd4000: add connection 192.168.202.102@tcp at head 00010000:00010000:1.0:1788572575.284951:0:20684:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff93a244fd4000-5 is initialized 00000020:00000080:1.0:1788572575.285198:0:20684:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff93a244fd4000 (uuid 96693705-5e29-4404-974e-e904d01a815e) 00000020:01000000:1.0:1788572575.285214:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff93a244fd4000 00000020:00000080:1.0:1788572575.285218:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:1.0:1788572575.285269:0:20684:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:1.0:1788572575.285279:0:20684:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff93a244fd4000: Adding target lustre-OST0000_UUID 00020000:01000000:1.0:1788572575.285284:0:20684:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:1.0:1788572575.285303:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572575.285307:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285315:0:20684:0:(obd_config.c:1563:class_process_config()) marker 13 (0x2) lustre-OST0000 add osc 00000020:01000000:1.0:1788572575.285319:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:1.0:1788572575.285322:0:20684:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #15 00000020:01000000:1.0:1788572575.285324:0:20684:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572575.285325:0:20684:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572575.285326:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:1.0:1788572575.285328:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285330:0:20684:0:(obd_config.c:1563:class_process_config()) marker 15 (0x6) lustre sys.timeout 00000020:01000000:1.0:1788572575.285332:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:1.0:1788572575.285337:0:20684:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.102@tcp:/lustre 00000020:00000080:1.0:1788572575.285340:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285341:0:20684:0:(obd_config.c:1563:class_process_config()) marker 19 (0x1) lustre-OST0001 add osc 00000020:00000080:1.0:1788572575.285343:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572575.285348:0:20684:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.102@tcp to nid 0x20000c0a8ca66 (192.168.202.102@tcp) 00000020:01000000:1.0:1788572575.285377:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff93a244fd4000 00000020:00000080:1.0:1788572575.285380:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572575.285389:0:20684:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff93a244fd4000 (ffff93a2481d19c0) 00000020:00000080:1.0:1788572575.285405:0:20684:0:(obd_config.c:720:class_attach_name()) OBD: dev 10 attached type osc with refcount 2 00000020:01000000:1.0:1788572575.285410:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff93a244fd4000 00000020:00000080:1.0:1788572575.285412:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572575.285501:0:20684:0:(ldlm_lib.c:102:import_set_conn()) imp ffff93a258ddd000@lustre-OST0001-osc-ffff93a244fd4000: add connection 192.168.202.102@tcp at head 00010000:00010000:1.0:1788572575.285703:0:20684:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff93a244fd4000-6 is initialized 00000020:00000080:1.0:1788572575.285884:0:20684:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff93a244fd4000 (uuid 96693705-5e29-4404-974e-e904d01a815e) 00000020:01000000:1.0:1788572575.285890:0:20684:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff93a244fd4000 00000020:00000080:1.0:1788572575.285892:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:1.0:1788572575.285901:0:20684:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:1.0:1788572575.285906:0:20684:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff93a244fd4000: Adding target lustre-OST0001_UUID 00020000:01000000:1.0:1788572575.285908:0:20684:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:1.0:1788572575.285915:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572575.285918:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285919:0:20684:0:(obd_config.c:1563:class_process_config()) marker 19 (0x2) lustre-OST0001 add osc 00000020:01000000:1.0:1788572575.285921:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572575.285928:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285930:0:20684:0:(obd_config.c:1563:class_process_config()) marker 21 (0x1) lustre sys.timeout 00000020:00000080:1.0:1788572575.285933:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:1.0:1788572575.285934:0:20684:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:1.0:1788572575.285937:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572575.285939:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285940:0:20684:0:(obd_config.c:1563:class_process_config()) marker 21 (0x2) lustre sys.timeout 00000020:01000000:1.0:1788572575.285942:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572575.285944:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.285946:0:20684:0:(obd_config.c:1563:class_process_config()) marker 25 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788572575.285948:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788572575.285999:0:20684:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20684:1875453495412352:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572575.286107:0:20684:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2495ed780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572575.293416:0:20684:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20684:1875453495412352:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572575.293741:0:20684:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572575.293751:0:20684:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572575.293754:0:20684:0:(obd_config.c:1563:class_process_config()) marker 25 (0x2) lustre sys.jobid_var 00000040:00080000:2.0:1788572575.293765:0:20684:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 42 count 42 00000020:01000000:0.0:1788572575.293886:0:20679:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-41 (rc=0) 10000000:01000000:0.0:1788572575.293912:0:20679:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572575.293939:0:20679:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d37 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.293967:0:20679:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a2742ad200/0x6d478a807bb11727 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d37 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572575.293975:0:20679:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff93a244fd4000 from 1 10000000:01000000:0.0:1788572575.293978:0:20679:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572575.294018:0:20679:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.294026:0:20679:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.294036:0:20679:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572575.294059:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412480:192.168.202.102@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572575.294136:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.297847:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412480:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572575.297889:0:20679:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d3e expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.297902:0:20679:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d3e expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572575.298484:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412608:192.168.202.102@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788572575.298591:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.317937:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412608:192.168.202.102@tcp:256:mount.lustre.0 10000000:01000000:3.0:1788572575.319668:0:20679:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:3.0:1788572575.319793:0:20679:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d3e expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.319808:0:20679:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a2742ac000/0x6d478a807bb11751 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d3e expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572575.319926:0:20679:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff93a244fd4000 from 1 10000000:01000000:2.0:1788572575.319929:0:20679:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572575.319980:0:20679:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572575.319990:0:20679:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572575.320002:0:20679:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572575.320035:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412736:192.168.202.102@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572575.320172:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.323994:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412736:192.168.202.102@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572575.324029:0:20679:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.324044:0:20679:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572575.324128:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412864:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572575.324211:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.327482:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412864:192.168.202.102@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572575.327624:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412992:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572575.328097:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.331682:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2f28dd17-6644-45dd-bf11-f8af88abec3c:20679:1875453495412992:192.168.202.102@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572575.332136:0:20688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20688:1875453495413120:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572575.332263:0:20688:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.336215:0:20688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20688:1875453495413120:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572575.336296:0:20688:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572575.336308:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.336310:0:20688:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:3.0:1788572575.336317:0:20688:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff93a244fd4000 00000020:00000080:3.0:1788572575.336319:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572575.336327:0:20688:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572575.336329:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.336331:0:20688:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:3.0:1788572575.336333:0:20688:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572575.336335:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.336337:0:20688:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:3.0:1788572575.336340:0:20688:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff93a244fd4000 00000020:00000080:3.0:1788572575.336342:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572575.336384:0:20688:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572575.336388:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.336389:0:20688:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:3.0:1788572575.336392:0:20688:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572575.336394:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572575.336395:0:20688:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre mdt.lustre-MDT0000.hsm_control 00000020:01000000:3.0:1788572575.336399:0:20688:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: lustre-ffff93a244fd4000 00000020:00000080:3.0:1788572575.336401:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000100:00100000:3.0:1788572575.336453:0:20688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20688:1875453495413248:192.168.202.102@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572575.336568:0:20688:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572575.349839:0:20688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:2f28dd17-6644-45dd-bf11-f8af88abec3c:20688:1875453495413248:192.168.202.102@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572575.349885:0:20688:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572575.349893:0:20688:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572575.349895:0:20688:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre mdt.lustre-MDT0000.hsm_control 00000040:00080000:1.0:1788572575.349901:0:20688:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:2.0:1788572575.350070:0:20679:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:2.0:1788572575.350088:0:20679:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788572575.350105:0:20679:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572575.350118:0:20679:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:2.0:1788572575.350140:0:20679:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:2.0:1788572575.350164:0:20679:0:(genops.c:1385:class_connect()) connect: client 96693705-5e29-4404-974e-e904d01a815e, cookie 0x6d478a807bb1175f 00800000:01000000:2.0:1788572575.350188:0:20679:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 96693705-5e29-4404-974e-e904d01a815e to lustre-clilmv-ffff93a244fd4000 00800000:01000000:2.0:1788572575.350196:0:20679:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff93a244fd4000(96693705-5e29-4404-974e-e904d01a815e) - lustre-MDT0000_UUID, 96693705-5e29-4404-974e-e904d01a815e 00000020:00000080:2.0:1788572575.350205:0:20679:0:(genops.c:1385:class_connect()) connect: client 96693705-5e29-4404-974e-e904d01a815e, cookie 0x6d478a807bb11766 00000100:00080000:2.0:1788572575.350214:0:20679:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:2.0:1788572575.350224:0:20679:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572575.350263:0:20679:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572575.350291:0:20679:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495413376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572575.350311:0:20679:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID 00800000:01000000:2.0:1788572575.350393:0:20679:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff93a244fd4000(96693705-5e29-4404-974e-e904d01a815e) successfully (5) 00800000:00000004:2.0:1788572575.350397:0:20679:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff93a244fd4000: age -204, max_age 0 00000080:00000004:2.0:1788572575.350423:0:20679:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff93a244fd4000: age -204, max_age 795 00800000:00000004:2.0:1788572575.350428:0:20679:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff93a244fd4000: age -204, max_age 795 00000100:00080000:2.0:1788572575.350443:0:20679:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff93a24940b100 x1875453495413632/t0(0) o41->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572575.350453:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062480 going to sleep for 0 seconds 00000100:00100000:2.0:1788572575.350511:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495413376:192.168.202.102@tcp:38:mount.lustre.0 00000100:00080000:2.0:1788572575.350621:0:2356:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff93a249408380 x1875453495413504/t0(0) o41->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572575.363382:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a244fd4000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572575.363401:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to target with instance 4 00000100:00080000:3.0:1788572575.363412:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff93a244fd4000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572575.363433:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:3.0:1788572575.363437:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:3.0:1788572575.363725:0:2352:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000d3ec4170): req@ffff93a24940b100 x1875453495413632/t0(0) o41->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572575.363751:0:2352:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000004ca58506): req@ffff93a249408380 x1875453495413504/t0(0) o41->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572575.363768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495413376:192.168.202.102@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572575.364436:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.368698:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:96693705-5e29-4404-974e-e904d01a815e:20679:1875453495413632:192.168.202.102@tcp:41:mount.lustre.0 00800000:00000004:3.0:1788572575.368725:0:20679:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff93a244fd4000: update ffff93a244b7d198 cache blocks 17239/17271 objects 531669/532011 00000080:00000004:3.0:1788572575.368732:0:20679:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff93a244fd4000: update ffff93a2516a0458 cache blocks 17239/17271 objects 531669/532011 00000100:00100000:3.0:1788572575.368773:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:96693705-5e29-4404-974e-e904d01a815e:20679:1875453495413760:192.168.202.102@tcp:40:mount.lustre.0 00000100:00100000:3.0:1788572575.368832:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572575.373693:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:96693705-5e29-4404-974e-e904d01a815e:20679:1875453495413760:192.168.202.102@tcp:40:mount.lustre.0 00000080:00000004:1.0:1788572575.373730:0:20679:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:1.0:1788572575.373750:0:20679:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 3407872 00020000:01000000:1.0:1788572575.373754:0:20679:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:1.0:1788572575.373768:0:20679:0:(genops.c:1385:class_connect()) connect: client 96693705-5e29-4404-974e-e904d01a815e, cookie 0x6d478a807bb1176d 00000020:00000080:1.0:1788572575.373791:0:20679:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x6d478a807bb11774 00000100:00080000:1.0:1788572575.373799:0:20679:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:1.0:1788572575.373807:0:20679:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a244fd4000: import ffff93a24a13b800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572575.373820:0:20679:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572575.373846:0:20679:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495413888/t0(0) o8->lustre-OST0000-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572575.373884:0:20679:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0000_UUID 00020000:01000000:1.0:1788572575.373901:0:20679:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff93a244fd4000) active 00000020:00000080:1.0:1788572575.373926:0:20679:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x6d478a807bb1177b 00000100:00080000:1.0:1788572575.373932:0:20679:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 0 00000100:00080000:1.0:1788572575.373937:0:20679:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff93a244fd4000: import ffff93a258ddd000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572575.373942:0:20679:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572575.373953:0:20679:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495414016/t0(0) o8->lustre-OST0001-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572575.373962:0:20679:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0001_UUID 00020000:01000000:1.0:1788572575.373967:0:20679:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff93a244fd4000) active 00000100:00100000:1.0:1788572575.374002:0:20679:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:96693705-5e29-4404-974e-e904d01a815e:20679:1875453495414144:192.168.202.102@tcp:33:mount.lustre.0 00000100:00100000:1.0:1788572575.374114:0:20679:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2495ed0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572575.374134:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495413888:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572575.374167:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495414016:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572575.374826:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:0:1875453495413504:192.168.202.102@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572575.385839:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a244fd4000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572575.385863:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff93a244fd4000: connect to target with instance 4 00000100:00080000:1.0:1788572575.385876:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff93a244fd4000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572575.385895:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788572575.385899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788572575.385907:0:2352:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:1.0:1788572575.385913:0:2352:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572575.385918:0:2352:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788572575.385953:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495413888:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572575.385970:0:2352:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff93a244fd4000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572575.385975:0:2352:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff93a244fd4000: connect to target with instance 4 00000100:00080000:1.0:1788572575.385981:0:2352:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff93a244fd4000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572575.385985:0:2352:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572575.385988:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258ddd000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788572575.385992:0:2352:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:1.0:1788572575.385995:0:2352:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572575.385997:0:2352:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788572575.386002:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495414016:192.168.202.102@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572575.386053:0:20679:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:96693705-5e29-4404-974e-e904d01a815e:20679:1875453495414144:192.168.202.102@tcp:33:mount.lustre.0 00000080:00200000:1.0:1788572575.386215:0:20679:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251393248) 00000080:00200000:1.0:1788572575.386225:0:20679:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:02000400:1.0:1788572575.386316:0:20679:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_4_g27fe0b7 00000080:00000004:1.0:1788572575.386321:0:20679:0:(super25.c:182:lustre_fill_super()) 192.168.202.102@tcp:/lustre: Mount complete 00000080:00200008:3.0:1788572575.587566:0:20709:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251393248), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572575.587581:0:20709:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251393248),name=/ 00000002:00010000:3.0:1788572575.587611:0:20709:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572575.587693:0:20709:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 20709 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.587706:0:20709:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 20709 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.587719:0:20709:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 20709 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572575.587762:0:20709:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f800 pname:cluuid:pid:xid:nid:opc:job mkdir:96693705-5e29-4404-974e-e904d01a815e:20709:1875453495414272:192.168.202.102@tcp:101:mkdir.0 00000100:00100000:3.0:1788572575.587864:0:20709:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258464d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572575.595090:0:20709:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f800 pname:cluuid:pid:xid:nid:opc:job mkdir:96693705-5e29-4404-974e-e904d01a815e:20709:1875453495414272:192.168.202.102@tcp:101:mkdir.0 00010000:00010000:3.0:1788572575.595120:0:20709:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595132:0:20709:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595145:0:20709:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595159:0:20709:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595469:0:20709:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595478:0:20709:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595485:0:20709:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572575.595497:0:20709:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064f800 x1875453495414272/t0(0) o101->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572591 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572575.595533:0:20709:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595546:0:20709:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a247a3d400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572575.595552:0:20709:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572575.595582:0:20709:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:3.0:1788572575.595588:0:20709:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251393248) for lock 0x6d478a807bb11782 00000080:00010000:3.0:1788572575.595593:0:20709:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251393248) 00000080:00010000:3.0:1788572575.595597:0:20709:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11782 from it ffffbaf6c1d5fcb8 00010000:00010000:3.0:1788572575.595600:0:20709:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572575.595609:0:20709:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572576.025006:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572577.051260:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572578.078169:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572579.097463:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572579.273010:0:20899:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Using TIMEOUT=20 00010000:00010000:3.0:1788572580.120456:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572580.701282:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2355:1875453495414400:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.708892:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572580.708919:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495414528:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.709802:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2355:1875453495414400:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.710422:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495414784:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.710492:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495415040:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.710545:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788572580.710552:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495414656:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.710575:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495414912:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.710593:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495415168:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.719912:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495414528:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.721438:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495414656:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.721457:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495414912:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.721469:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495415168:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.721480:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495414784:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.721491:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495415040:192.168.202.102@tcp:400:kworker.0 00010000:00010000:0.0:1788572581.142715:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572582.167388:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572583.190469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572583.469571:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572583.469615:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572583.469711:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518461056 00000100:00100000:0.0:1788572583.469721:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518461056 00000100:00100000:0.0:1788572583.469760:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518461184 00000100:00100000:0.0:1788572583.469763:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518461184 00000100:00100000:0.0:1788572583.469777:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 0 00000100:00100000:0.0:1788572583.469785:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2f28dd17-6644-45dd-bf11-f8af88abec3c+4:25771:x1875453518461056:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572583.469803:0:20550:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.469885:0:20550:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572583.469913:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2f28dd17-6644-45dd-bf11-f8af88abec3c+4:25771:x1875453518461056:12345-192.168.202.102@tcp:104: Request processed in 129us (353us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572583.469921:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 0 00000100:00100000:0.0:1788572583.469938:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 1 00000100:00100000:0.0:1788572583.469945:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2f28dd17-6644-45dd-bf11-f8af88abec3c+3:25771:x1875453518461184:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572583.469953:0:20550:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.469995:0:20550:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:0.0:1788572583.470001:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:2f28dd17-6644-45dd-bf11-f8af88abec3c+3:25771:x1875453518461184:12345-192.168.202.102@tcp:104: Request processed in 57us (385us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572583.470007:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 1 00010000:00010000:0.0:1788572583.470024:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788572583.470029:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1400000000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470037:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217800 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:0.0:1788572583.470041:0:20561:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1400400000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470049:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1408400000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572583.470066:0:20561:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1409400000000 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572583.470071:0:20561:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788572583.470109:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470124:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572583.470154:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470160:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a243217800/0x6d478a807bb11758 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5c09400000020 nid: local remote: 0x977d9e7969c01d45 expref: -99 pid: 20679 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470176:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572583.470180:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470185:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216400 already unused, calling callback (ffffffffc1341c20) 10000000:00010000:0.0:1788572583.470188:0:20561:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470195:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572583.470201:0:20561:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572583.470205:0:20561:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788572583.470219:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470224:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572583.470270:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572583.470378:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572583.470676:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.102@tcp lock: ffff93a243216400/0x6d478a807bb116f6 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x977d9e7969c01d0d expref: -99 pid: 20549 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572583.470702:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572583.470703:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572583.470705:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572583.470731:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495415296:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572583.470792:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572583.470798:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495415424:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572583.475823:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495415424:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572583.475838:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495415296:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572584.214962:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572584.855430:0:20563:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788572584.855441:0:20563:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff93a244fd4000 from 10 10000000:01000000:2.0:1788572584.855449:0:20563:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572584.855519:0:20563:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.855533:0:20563:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.855546:0:20563:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572584.855581:0:20563:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415552:192.168.202.102@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.855690:0:20563:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572584.861128:0:20563:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415552:192.168.202.102@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572584.861158:0:20563:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d76 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.861171:0:20563:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d76 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572584.861439:0:20563:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415680:192.168.202.102@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.861607:0:20563:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572584.865919:0:20563:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415680:192.168.202.102@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.866005:0:20563:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415808:192.168.202.102@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.866109:0:20563:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572584.872033:0:20563:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415808:192.168.202.102@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572584.878970:0:21158:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 10 00000020:01000000:2.0:1788572584.879155:0:20563:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 10-9 (rc=0) 10000000:01000000:2.0:1788572584.879174:0:20563:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788572584.879185:0:20563:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d76 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.879197:0:20563:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a243216800/0x6d478a807bb11789 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d76 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572584.879207:0:20563:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788572584.879209:0:20563:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff93a251f1d800 from 10 10000000:01000000:2.0:1788572584.879212:0:20563:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572584.879287:0:20563:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.879296:0:20563:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.879306:0:20563:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572584.879436:0:20563:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415936:192.168.202.102@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.879526:0:20563:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572584.881940:0:20563:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495415936:192.168.202.102@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572584.881971:0:20563:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d8b expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.881983:0:20563:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x977d9e7969c01d8b expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572584.882053:0:20563:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495416064:192.168.202.102@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.882174:0:20563:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572584.885764:0:20563:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495416064:192.168.202.102@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.885825:0:20563:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495416192:192.168.202.102@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788572584.885917:0:20563:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517da80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572584.892405:0:20563:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:2f28dd17-6644-45dd-bf11-f8af88abec3c:20563:1875453495416192:192.168.202.102@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788572584.895600:0:21160:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 10 00000020:01000000:2.0:1788572584.900123:0:20563:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 10-9 (rc=0) 10000000:01000000:2.0:1788572584.900172:0:20563:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.102@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788572584.900185:0:20563:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d8b expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572584.900196:0:20563:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.102@tcp lock: ffff93a243217200/0x6d478a807bb11790 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x977d9e7969c01d8b expref: -99 pid: 20563 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:0.0:1788572584.956188:0:21167:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572584.956202:0:21167:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572584.956345:0:21167:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572584.956367:0:21167:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572584.956383:0:21167:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572584.956392:0:21167:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572584.956400:0:21167:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572584.956405:0:21167:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:0.0:1788572584.956412:0:21167:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f27cb8 00010000:00010000:0.0:1788572584.956417:0:21167:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572584.956440:0:21167:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200008:0.0:1788572584.957129:0:21167:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788572584.957136:0:21167:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572584.957147:0:21167:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572584.957158:0:21167:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572584.957169:0:21167:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572584.957176:0:21167:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572584.957180:0:21167:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572584.957183:0:21167:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:0.0:1788572584.957187:0:21167:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f27cb8 00010000:00010000:0.0:1788572584.957191:0:21167:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572584.957198:0:21167:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788572584.957480:0:21167:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:1.0:1788572584.957494:0:21167:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572584.957503:0:21167:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572584.957513:0:21167:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788572584.957531:0:21167:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572584.957536:0:21167:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:1.0:1788572584.957539:0:21167:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:1.0:1788572584.957544:0:21167:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f27bb0 00010000:00010000:1.0:1788572584.957547:0:21167:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572584.957554:0:21167:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572584.957563:0:21167:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572584.957571:0:21167:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 100000 00000080:00200000:3.0:1788572584.960188:0:21168:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572584.960220:0:21168:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572584.960267:0:21168:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572584.960285:0:21168:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572584.960293:0:21168:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572584.960306:0:21168:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572584.960309:0:21168:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572584.960316:0:21168:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f3fbb0 00010000:00010000:3.0:1788572584.960321:0:21168:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572584.960330:0:21168:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572584.960343:0:21168:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 44 00000080:00200008:3.0:1788572584.960349:0:21168:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 100000 00000080:00200000:3.0:1788572584.960369:0:21168:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572584.960375:0:21168:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572584.960382:0:21168:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572584.960390:0:21168:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572584.960396:0:21168:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572584.960400:0:21168:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572584.960403:0:21168:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572584.960406:0:21168:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f3fbf8 00010000:00010000:3.0:1788572584.960409:0:21168:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572584.960417:0:21168:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572584.960437:0:21168:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572584.960446:0:21168:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=.lustre, dir=[0x200000007:0x1:0x0](ffff93a251173a88), flags=17 00000080:00200000:3.0:1788572584.960451:0:21168:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=.lustre, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=0 00000002:00010000:3.0:1788572584.960462:0:21168:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .lustre,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788572584.960499:0:21168:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21168 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.960506:0:21168:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21168 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.960515:0:21168:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21168 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572584.960536:0:21168:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21168:1875453495416320:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:3.0:1788572584.960618:0:21168:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572584.965897:0:21168:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21168:1875453495416320:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:3.0:1788572584.965931:0:21168:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.965950:0:21168:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 4/1,0 mode: --/PR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.965957:0:21168:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 4/1,0 mode: --/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.965968:0:21168:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.965986:0:21168:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.965995:0:21168:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 5/2,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.966001:0:21168:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 5/2,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.966005:0:21168:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572584.966013:0:21168:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff93a25064f800 x1875453495416320/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572600 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572584.966022:0:21168:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.966030:0:21168:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589934594/1 (0 0) 00000080:00200000:3.0:1788572584.966034:0:21168:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572584.966057:0:21168:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000002:0x1:0x0](ffff93a251109148) 00000080:00200000:3.0:1788572584.966062:0:21168:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000002:0x1:0x0], updating i_size 16384 00000080:00010000:3.0:1788572584.966067:0:21168:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000002:0x1:0x0](ffff93a251109148) for lock 0x6d478a807bb11797 00000080:00010000:3.0:1788572584.966075:0:21168:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000002:0x1:0x0](ffff93a251109148) 00000080:00010000:3.0:1788572584.966077:0:21168:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11797 from it ffffbaf6c1f3fac8 00010000:00010000:3.0:1788572584.966080:0:21168:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.966108:0:21168:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9000/0x6d478a807bb11797 lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01dc3 expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572584.966146:0:21168:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x1:0x0](ffff93a251109148), inode mode 41ed mask 1 00000080:00200000:3.0:1788572584.966165:0:21168:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=fid, dir=[0x200000002:0x1:0x0](ffff93a251109148), file ffff93a251663b00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1788572584.966170:0:21168:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fid, dir=[0x200000002:0x1:0x0](ffff93a251109148), intent=open 00000002:00010000:3.0:1788572584.966188:0:21168:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fid,[0x0:0x0:0x0]) in obj [0x200000002:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1788572584.966257:0:21168:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 3/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21168 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.966264:0:21168:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 3/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21168 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572584.966275:0:21168:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572584.966281:0:21168:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064f800 x1875453495416576/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572584.966290:0:21168:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 3/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21168 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572584.966308:0:21168:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21168:1875453495416576:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:3.0:1788572584.966465:0:21168:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572584.972860:0:21168:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21168:1875453495416576:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:3.0:1788572584.972895:0:21168:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 4/1,0 mode: --/CR res: [0x200000002:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972919:0:21168:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 4/1,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972939:0:21168:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 4/1,0 mode: --/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972951:0:21168:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972963:0:21168:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972978:0:21168:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 5/2,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972988:0:21168:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 5/2,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.972995:0:21168:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 4/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572584.973007:0:21168:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064f800 x1875453495416576/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572600 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572584.973021:0:21168:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064f800 x1875453495416576/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572600 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572584.973031:0:21168:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.973041:0:21168:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589934594/2 (0 0) 00000080:00200000:3.0:1788572584.973054:0:21168:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572584.973102:0:21168:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110c2c8) 00000080:00200000:3.0:1788572584.973110:0:21168:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000002:0x2:0x0], updating i_size 16384 00000080:00010000:3.0:1788572584.973116:0:21168:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000002:0x2:0x0](ffff93a25110c2c8) for lock 0x6d478a807bb1179e 00000080:00010000:3.0:1788572584.973162:0:21168:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000002:0x2:0x0](ffff93a25110c2c8) 00000080:00200008:3.0:1788572584.973179:0:21168:0:(file.c:1090:ll_file_open()) START file fid:[0x200000002:0x2:0x0](ffff93a25110c2c8), flags 100000 00000002:00100000:3.0:1788572584.973198:0:21168:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064f800 x1875453495416576/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572600 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572584.973266:0:21168:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1179e from it ffff93a250f7acc0 00010000:00010000:3.0:1788572584.973271:0:21168:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 3/1,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572584.973281:0:21168:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8e00/0x6d478a807bb1179e lrc: 2/0,0 mode: PR/PR res: [0x200000002:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ddf expref: -99 pid: 21168 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572584.973301:0:21168:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110c2c8), inode mode 4040 mask 44 00000080:00200080:3.0:1788572584.973374:0:21168:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=401866d5 arg=c3e980 00800000:00000080:3.0:1788572584.973387:0:21168:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d5 len=28 karg=ffff93a247a388a0 uarg=0000000000000000 00000002:00000080:3.0:1788572584.973397:0:21168:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d5 len=28 karg=ffff93a247a388a0 uarg=0000000000000000 00000100:00100000:3.0:1788572584.973437:0:21168:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21168:1875453495416704:192.168.202.102@tcp:59:lhsmtool_posix.0 00000100:00100000:3.0:1788572584.973504:0:21168:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572584.977520:0:21168:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21168:1875453495416704:192.168.202.102@tcp:59:lhsmtool_posix.0 00000080:00200000:3.0:1788572585.096481:0:21176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.096513:0:21176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.096532:0:21176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572585.096547:0:21176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572585.096562:0:21176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.096571:0:21176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.096575:0:21176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572585.096582:0:21176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f1fbb0 00010000:00010000:3.0:1788572585.096597:0:21176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572585.096607:0:21176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572585.096622:0:21176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 44 00000080:00200008:3.0:1788572585.096631:0:21176:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 504000 00000080:00200080:3.0:1788572585.096652:0:21176:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=8080667f arg=7fff622c7520 00000080:00200008:3.0:1788572585.096670:0:21176:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 504000 00010000:00010000:2.0:1788572585.258940:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572585.274063:0:21182:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.274104:0:21182:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.274132:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572585.274149:0:21182:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572585.274158:0:21182:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.274166:0:21182:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.274169:0:21182:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572585.274177:0:21182:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f67b58 00010000:00010000:3.0:1788572585.274187:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572585.274197:0:21182:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572585.274214:0:21182:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.274247:0:21182:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), flags=13 00000080:00200000:3.0:1788572585.274253:0:21182:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=getattr 00000002:00010000:3.0:1788572585.274268:0:21182:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572585.274322:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.274331:0:21182:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.274341:0:21182:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.274370:0:21182:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21182:1875453495416832:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572585.274477:0:21182:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.277907:0:21182:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21182:1875453495416832:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572585.277933:0:21182:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.277949:0:21182:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.277955:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.277971:0:21182:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.277978:0:21182:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.277983:0:21182:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be9600 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572585.277988:0:21182:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.277995:0:21182:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.278001:0:21182:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.278007:0:21182:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572585.278014:0:21182:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.278020:0:21182:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117a5 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21182 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572585.278038:0:21182:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff93a25064dc00 x1875453495416832/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572585.278053:0:21182:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572585.278088:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788572585.278101:0:21182:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00010000:3.0:1788572585.278124:0:21182:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f67930 00010000:00010000:3.0:1788572585.278129:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788572585.278137:0:21182:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1788572585.278218:0:21182:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.278252:0:21182:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.278260:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788572585.278269:0:21182:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1788572585.278275:0:21182:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.278280:0:21182:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.278283:0:21182:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572585.278288:0:21182:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f67be8 00010000:00010000:3.0:1788572585.278291:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788572585.278298:0:21182:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788572585.278306:0:21182:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.278314:0:21182:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=16392 00000080:00200008:3.0:1788572585.278343:0:21182:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572585.278348:0:21182:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.278356:0:21182:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.278363:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788572585.278372:0:21182:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788572585.278378:0:21182:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.278382:0:21182:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.278385:0:21182:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572585.278388:0:21182:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f67cb8 00010000:00010000:3.0:1788572585.278392:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788572585.278399:0:21182:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200008:3.0:1788572585.279480:0:21182:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572585.279489:0:21182:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.279501:0:21182:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.279516:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572585.279529:0:21182:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788572585.279536:0:21182:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.279546:0:21182:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.279549:0:21182:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:3.0:1788572585.279554:0:21182:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f67cb8 00010000:00010000:3.0:1788572585.279558:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572585.279567:0:21182:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572585.286924:0:21182:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572585.286961:0:21182:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572585.286983:0:21182:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572585.287000:0:21182:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572585.287023:0:21182:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572585.287033:0:21182:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572585.287038:0:21182:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb11720 00000080:00010000:2.0:1788572585.287047:0:21182:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11720 from it ffffbaf6c1f67bb0 00010000:00010000:2.0:1788572585.287052:0:21182:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572585.287063:0:21182:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788572585.287096:0:21182:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572585.287110:0:21182:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 300000 00000080:00200080:2.0:1788572585.287993:0:21182:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c03066f0 arg=7ffce906d1a0 00000080:00200000:2.0:1788572585.288011:0:21182:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) name=d1f.sanity-pcc stripe_offset=0 stripe_count=1, hash_type=0 00000100:00100000:2.0:1788572585.288099:0:21182:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21182:1875453495416960:192.168.202.102@tcp:700:lfs.0 00000100:00100000:2.0:1788572585.288549:0:21182:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517db40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572585.302554:0:21182:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21182:1875453495416960:192.168.202.102@tcp:700:lfs.0 00010000:00010000:2.0:1788572585.302668:0:21182:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788572585.308132:0:21182:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788572585.308151:0:21182:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572585.308154:0:21182:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217400/0x6d478a807bb11720 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01d30 expref: -99 pid: 20593 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000020:00100000:3.0:1788572585.308194:0:21182:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572585.308207:0:21182:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064dc00 x1875453495417216/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572585.308270:0:21182:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21182:1875453495417216:192.168.202.102@tcp:36:lfs.0 00000100:00100000:3.0:1788572585.308360:0:21182:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572585.311578:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572585.311702:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518463360 00000100:00100000:0.0:1788572585.311721:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518463360 00000100:00100000:0.0:1788572585.311794:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 2 00000100:00100000:0.0:1788572585.311804:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:96693705-5e29-4404-974e-e904d01a815e+5:23657:x1875453518463360:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572585.311843:0:20550:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572585.311936:0:20550:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572585.311943:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:96693705-5e29-4404-974e-e904d01a815e+5:23657:x1875453518463360:12345-192.168.202.102@tcp:104: Request processed in 141us (380us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572585.311950:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 2 00010000:00010000:2.0:1788572585.312204:0:20560:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572585.312213:0:20560:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312220:0:20560:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be9a00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572585.312269:0:20560:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312279:0:20560:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312287:0:20560:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312339:0:20560:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312347:0:20560:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572585.312374:0:20560:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312382:0:20560:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a244fd4000 lock: ffff93a253be9a00/0x6d478a807bb11782 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c01d68 expref: -99 pid: 20709 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.312402:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572585.312404:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572585.312406:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572585.312413:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572585.312427:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96693705-5e29-4404-974e-e904d01a815e:2355:1875453495417344:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572585.316052:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:96693705-5e29-4404-974e-e904d01a815e:2355:1875453495417344:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572585.343906:0:21182:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21182:1875453495417216:192.168.202.102@tcp:36:lfs.0 00000080:00200000:3.0:1788572585.343960:0:21182:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00200000:3.0:1788572585.343968:0:21182:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200008:3.0:1788572585.344005:0:21182:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), flags 300000 00000080:00200000:1.0:1788572585.386321:0:21183:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:1.0:1788572585.391971:0:21183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572585.392045:0:21183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 21183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572585.392057:0:21183:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572585.392078:0:21183:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21183 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572585.392106:0:21183:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job chmod:bf89ae46-999b-46d3-9a62-08dfe43909d0:21183:1875453495417472:192.168.202.102@tcp:101:chmod.0 00000100:00100000:1.0:1788572585.392204:0:21183:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a253e82e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572585.399498:0:21183:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job chmod:bf89ae46-999b-46d3-9a62-08dfe43909d0:21183:1875453495417472:192.168.202.102@tcp:101:chmod.0 00010000:00010000:2.0:1788572585.399532:0:21183:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399557:0:21183:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399570:0:21183:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399581:0:21183:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399789:0:21183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399798:0:21183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399804:0:21183:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572585.399816:0:21183:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a258da0a80 x1875453495417472/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572585.399831:0:21183:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.399843:0:21183:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572585.399849:0:21183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572585.400402:0:21183:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:2.0:1788572585.400419:0:21183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:2.0:1788572585.400425:0:21183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572585.400427:0:21183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f4fb58 00010000:00010000:2.0:1788572585.400432:0:21183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.400441:0:21183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572585.400469:0:21183:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572585.400492:0:21183:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), flags=9 00000080:00200000:2.0:1788572585.400497:0:21183:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=getattr 00000002:00010000:2.0:1788572585.400518:0:21183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572585.400565:0:21183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.400573:0:21183:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.400582:0:21183:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21183 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572585.400716:0:21183:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job chmod:bf89ae46-999b-46d3-9a62-08dfe43909d0:21183:1875453495417600:192.168.202.102@tcp:101:chmod.0 00000100:00100000:2.0:1788572585.401095:0:21183:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517db40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.407619:0:21183:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job chmod:bf89ae46-999b-46d3-9a62-08dfe43909d0:21183:1875453495417600:192.168.202.102@tcp:101:chmod.0 00010000:00010000:3.0:1788572585.407648:0:21183:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407682:0:21183:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407691:0:21183:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407702:0:21183:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407712:0:21183:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407725:0:21183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407734:0:21183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.407741:0:21183:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572585.407754:0:21183:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a24940b480 x1875453495417600/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572585.408902:0:21183:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.408925:0:21183:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/1 (0 0) 00000080:00200000:3.0:1788572585.408942:0:21183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572585.408964:0:21183:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200000:3.0:1788572585.408968:0:21183:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x1:0x0](ffff93a25110ec08): rc = 0 00000080:00010000:3.0:1788572585.408973:0:21183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117b3 00000080:00010000:3.0:1788572585.408984:0:21183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572585.408987:0:21183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117b3 from it ffffbaf6c1f4fb20 00010000:00010000:3.0:1788572585.408992:0:21183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.409000:0:21183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572585.409039:0:21183:0:(file.c:6296:ll_getattr_dentry()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572585.409046:0:21183:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08),name=d1f.sanity-pcc 00000002:00010000:3.0:1788572585.409075:0:21183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.409085:0:21183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.409095:0:21183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572585.409101:0:21183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.409106:0:21183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572585.409119:0:21183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117b3 00000080:00010000:3.0:1788572585.409123:0:21183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117b3 from it ffffbaf6c1f4fcb8 00010000:00010000:3.0:1788572585.409126:0:21183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.409144:0:21183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572585.409526:0:21183:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.409535:0:21183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.409544:0:21183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.409553:0:21183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572585.409560:0:21183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.409565:0:21183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.409567:0:21183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572585.409571:0:21183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f4fc08 00010000:00010000:3.0:1788572585.409575:0:21183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.409583:0:21183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572585.409591:0:21183:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.409597:0:21183:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=9 00000080:00200008:3.0:1788572585.409609:0:21183:0:(llite_lib.c:2427:ll_setattr_raw()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08) current size 16384, valid attrs 41, mode 41ff, uid 1243279360, gid -27742, new size 0, atime 1788572585.000000000, mtime 1788572585.000000000, ctime 1788572585.000000000 00010000:00010000:3.0:1788572585.409626:0:21183:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572585.409681:0:21183:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572585.409689:0:21183:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572585.409692:0:21183:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216000/0x6d478a807bb117b3 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01e33 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788572585.409714:0:21183:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572585.409725:0:21183:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064ed80 x1875453495417856/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572585.409747:0:21183:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job chmod:bf89ae46-999b-46d3-9a62-08dfe43909d0:21183:1875453495417856:192.168.202.102@tcp:36:chmod.0 00000100:00100000:3.0:1788572585.411060:0:21183:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.425305:0:21183:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job chmod:bf89ae46-999b-46d3-9a62-08dfe43909d0:21183:1875453495417856:192.168.202.102@tcp:36:chmod.0 00000080:00200000:3.0:1788572585.425349:0:21183:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200008:3.0:1788572585.425360:0:21183:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08) current size 16384, valid attrs 10000041, mode 41ff, uid 1243279360, gid -27742, new size 0, atime 1788572585.000000000, mtime 1788572585.000000000, ctime 1788572585.000000000, rc 0 00000080:00200000:3.0:1788572585.454416:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.454449:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.454469:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572585.454485:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572585.454492:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.454507:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.454511:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572585.454519:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f6fb58 00010000:00010000:3.0:1788572585.454523:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572585.454531:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572585.454547:0:21184:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.454561:0:21184:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), flags=8 00000080:00200000:3.0:1788572585.454566:0:21184:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=getattr 00000002:00010000:3.0:1788572585.454578:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572585.454625:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.454634:0:21184:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.454643:0:21184:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.454668:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495417984:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572585.454811:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.457643:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495417984:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572585.457671:0:21184:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457710:0:21184:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457718:0:21184:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457729:0:21184:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457739:0:21184:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457752:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457761:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457768:0:21184:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572585.457781:0:21184:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064c000 x1875453495417984/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572585.457795:0:21184:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457805:0:21184:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/1 (0 0) 00000080:00200000:3.0:1788572585.457817:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572585.457840:0:21184:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200000:3.0:1788572585.457843:0:21184:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x1:0x0](ffff93a25110ec08): rc = 0 00000080:00010000:3.0:1788572585.457849:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117ba 00000080:00010000:3.0:1788572585.457863:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572585.457867:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ba from it ffffbaf6c1f6fb20 00010000:00010000:3.0:1788572585.457871:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.457879:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572585.457921:0:21184:0:(file.c:6296:ll_getattr_dentry()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572585.457928:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08),name=d1f.sanity-pcc 00000002:00010000:3.0:1788572585.457948:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.457957:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.457969:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572585.457977:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.457981:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572585.457984:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117ba 00000080:00010000:3.0:1788572585.457988:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ba from it ffffbaf6c1f6fcb8 00010000:00010000:3.0:1788572585.457992:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.458010:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572585.458087:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.458096:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.458104:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572585.458121:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572585.458126:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.458130:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.458133:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572585.458136:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f6fb58 00010000:00010000:3.0:1788572585.458139:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572585.458146:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572585.458155:0:21184:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.458161:0:21184:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=8 00000080:00200008:3.0:1788572585.458167:0:21184:0:(file.c:6296:ll_getattr_dentry()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572585.458170:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08),name=d1f.sanity-pcc 00000002:00010000:3.0:1788572585.458177:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.458189:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572585.458196:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572585.458203:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.458207:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572585.458210:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117ba 00000080:00010000:3.0:1788572585.458214:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ba from it ffffbaf6c1f6fcb8 00010000:00010000:3.0:1788572585.458217:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572585.458223:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572585.458280:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.458287:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.458293:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572585.458301:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572585.458307:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.458311:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.458314:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572585.458317:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f6fbf8 00010000:00010000:3.0:1788572585.458325:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572585.458331:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572585.458339:0:21184:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.458343:0:21184:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=265 00000080:00200000:3.0:1788572585.458347:0:21184:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 44 00000080:00200008:3.0:1788572585.458355:0:21184:0:(file.c:1090:ll_file_open()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 104400 00000002:00010000:3.0:1788572585.458368:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240104401 00010000:00010000:3.0:1788572585.458409:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.458416:0:21184:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572585.458426:0:21184:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572585.458433:0:21184:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c000 x1875453495418240/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572585.458441:0:21184:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.458461:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495418240:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572585.458611:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.465693:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495418240:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572585.465718:0:21184:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465751:0:21184:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465758:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465765:0:21184:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465775:0:21184:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465782:0:21184:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be8200 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572585.465787:0:21184:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465793:0:21184:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465800:0:21184:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465806:0:21184:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572585.465813:0:21184:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.465820:0:21184:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117c1 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572585.465841:0:21184:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064c000 x1875453495418240/t17179869188(17179869188) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572585.465868:0:21184:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064c000 x1875453495418240/t17179869188(17179869188) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572585.465880:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572585.465900:0:21184:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000002:00100000:3.0:1788572585.465915:0:21184:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064c000 x1875453495418240/t17179869188(17179869188) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1788572585.465991:0:21184:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) cmd=801c581f arg=7ffecd2e4830 00000080:00200080:3.0:1788572585.466000:0:21184:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) cmd=401c5820 arg=7ffecd2e4830 00010000:00010000:3.0:1788572585.466021:0:21184:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572585.466079:0:21184:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572585.466086:0:21184:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572585.466089:0:21184:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117ba lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01e56 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572585.466116:0:21184:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572585.466122:0:21184:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064e680 x1875453495418496/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572585.466140:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495418496:192.168.202.102@tcp:36:lfs.0 00000100:00100000:3.0:1788572585.466204:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.472530:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495418496:192.168.202.102@tcp:36:lfs.0 00000080:00200008:3.0:1788572585.472576:0:21184:0:(file.c:479:ll_file_release()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 104000 00000002:00100000:3.0:1788572585.472643:0:21184:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064c000 x1875453495418240/t17179869188(17179869188) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572601 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572585.472655:0:21184:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572585.472661:0:21184:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064d880 x1875453495418752/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572585.472683:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495418752:192.168.202.102@tcp:35:lfs.0 00000100:00100000:3.0:1788572585.472743:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.483161:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495418752:192.168.202.102@tcp:35:lfs.0 00000002:00100000:3.0:1788572585.483211:0:21184:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064c000 x1875453495418240/t17179869188(17179869188) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572585.483406:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572585.483426:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.483447:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572585.483461:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572585.483469:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572585.483475:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572585.483479:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572585.483486:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f6fbf8 00010000:00010000:3.0:1788572585.483491:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572585.483500:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572585.483532:0:21184:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572585.483556:0:21184:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), file ffff93a253983980, open_flags 18800, mode 0 opened 0 00000080:00200000:3.0:1788572585.483562:0:21184:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=open 00000002:00010000:3.0:1788572585.483576:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:3.0:1788572585.483624:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.483632:0:21184:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572585.483642:0:21184:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572585.483649:0:21184:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064d880 x1875453495419008/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572585.483657:0:21184:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.483676:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419008:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572585.483836:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.487494:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419008:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572585.487520:0:21184:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487545:0:21184:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487552:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487576:0:21184:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487585:0:21184:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487591:0:21184:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a253be9600 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572585.487596:0:21184:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487603:0:21184:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487611:0:21184:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487617:0:21184:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572585.487624:0:21184:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487631:0:21184:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117c8 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572585.487652:0:21184:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064d880 x1875453495419008/t17179869191(17179869191) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572585.487665:0:21184:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064d880 x1875453495419008/t17179869191(17179869191) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572585.487677:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572585.487698:0:21184:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200000:3.0:1788572585.487707:0:21184:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x1:0x0](ffff93a25110ec08): rc = 0 00000080:00010000:3.0:1788572585.487711:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x0 00000080:00200008:3.0:1788572585.487733:0:21184:0:(file.c:1090:ll_file_open()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00100000:3.0:1788572585.487752:0:21184:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064d880 x1875453495419008/t17179869191(17179869191) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572585.487810:0:21184:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 44 00000080:00200008:3.0:1788572585.487834:0:21184:0:(file.c:6296:ll_getattr_dentry()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572585.487841:0:21184:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08),name=d1f.sanity-pcc 00000002:00010000:3.0:1788572585.487853:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572585.487906:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487913:0:21184:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.487923:0:21184:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21184 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.487942:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419136:192.168.202.102@tcp:101:lfs.0 00000100:00100000:3.0:1788572585.488013:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.492029:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419136:192.168.202.102@tcp:101:lfs.0 00010000:00010000:3.0:1788572585.492059:0:21184:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492082:0:21184:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492093:0:21184:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492129:0:21184:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492143:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492150:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492157:0:21184:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572585.492168:0:21184:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064c000 x1875453495419136/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:100 00000002:00010000:3.0:1788572585.492180:0:21184:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492483:0:21184:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/1 (0 0) 00000080:00200000:3.0:1788572585.492491:0:21184:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200001b71:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572585.492513:0:21184:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00010000:3.0:1788572585.492518:0:21184:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117cf 00000080:00010000:3.0:1788572585.492523:0:21184:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572585.492526:0:21184:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117cf from it ffffbaf6c1f6fcd8 00010000:00010000:3.0:1788572585.492529:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.492647:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572585.492909:0:21184:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) pos/size0/16384 32bit_api 0 00000002:00010000:3.0:1788572585.492929:0:21184:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200001b71:0x1:0x0], intent: readdir flags 0 00010000:00010000:3.0:1788572585.492939:0:21184:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.492950:0:21184:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788572585.495168:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419264:192.168.202.102@tcp:37:lfs.0 00000100:00100000:3.0:1788572585.495324:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.504546:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419264:192.168.202.102@tcp:37:lfs.0 00010000:00010000:3.0:1788572585.505187:0:21184:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572585.505205:0:21184:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572585.505279:0:21184:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) pos/size18446744073709551614/16384 32bit_api 0 00000080:00200008:3.0:1788572585.505397:0:21184:0:(file.c:479:ll_file_release()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00100000:3.0:1788572585.505455:0:21184:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064d880 x1875453495419008/t17179869191(17179869191) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572585.505466:0:21184:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572585.505484:0:21184:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c000 x1875453495419520/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:3.0:1788572585.505506:0:21184:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419520:192.168.202.102@tcp:35:lfs.0 00000100:00100000:3.0:1788572585.505574:0:21184:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.510936:0:21184:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21184:1875453495419520:192.168.202.102@tcp:35:lfs.0 00000002:00100000:3.0:1788572585.510972:0:21184:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064d880 x1875453495419008/t17179869191(17179869191) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572585.612995:0:21190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572585.613026:0:21190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572585.613045:0:21190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572585.613073:0:21190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572585.613083:0:21190:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:0.0:1788572585.613092:0:21190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572585.613096:0:21190:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:0.0:1788572585.613178:0:21190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f7fbf8 00010000:00010000:0.0:1788572585.613553:0:21190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572585.613563:0:21190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788572585.613579:0:21190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572585.613597:0:21190:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), flags=25 00000080:00200000:0.0:1788572585.613610:0:21190:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=0 00000002:00010000:0.0:1788572585.613622:0:21190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572585.613669:0:21190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21190 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.613676:0:21190:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21190 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.613686:0:21190:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21190 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572585.613792:0:21190:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1880 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21190:1875453495419648:192.168.202.102@tcp:101:dd.500 00000100:00100000:0.0:1788572585.614010:0:21190:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2429cb3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572585.644086:0:21190:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1880 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21190:1875453495419648:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572585.644327:0:21190:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644552:0:21190:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644560:0:21190:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644572:0:21190:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644582:0:21190:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644596:0:21190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644604:0:21190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644612:0:21190:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572585.644626:0:21190:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff93a2460d1880 x1875453495419648/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00010000:2.0:1788572585.644639:0:21190:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644650:0:21190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572585.644670:0:21190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572585.644679:0:21190:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644686:0:21190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644693:0:21190:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.644719:0:21190:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572585.644726:0:21190:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572585.644749:0:21190:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200000:2.0:1788572585.644753:0:21190:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x1:0x0](ffff93a25110ec08): rc = 0 00000080:00010000:2.0:1788572585.644758:0:21190:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117cf 00000080:00010000:2.0:1788572585.644770:0:21190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:2.0:1788572585.644774:0:21190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117cf from it ffffbaf6c1f7fac8 00010000:00010000:2.0:1788572585.644778:0:21190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572585.644787:0:21190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572585.645004:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572585.645012:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.645020:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742ade00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:0.0:1788572585.645026:0:20561:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x11 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.645034:0:20561:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x11 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.645043:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 3/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x11 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.645069:0:20561:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572585.645328:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x11 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.645336:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000080:00200000:2.0:1788572585.646093:0:21190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1788572585.646125:0:21190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1788572585.646138:0:21190:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), file ffff93a250cf3200, open_flags 8241, mode 81b6 opened 0 00000080:00200000:2.0:1788572585.646147:0:21190:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=open|creat 00000002:00010000:2.0:1788572585.646179:0:21190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1f.sanity-pcc,[0x200001b71:0x2:0x0]) in obj [0x200001b71:0x1:0x0], intent: open|creat flags 010000000101102 00010000:00010000:2.0:1788572585.646260:0:21190:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572585.646407:0:21190:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572585.646415:0:21190:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572585.646418:0:21190:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be8200/0x6d478a807bb117cf lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01e95 expref: -99 pid: 21184 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572585.646445:0:21190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21190 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.646452:0:21190:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21190 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572585.646463:0:21190:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572585.646473:0:21190:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940ad80 x1875453495420032/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:2.0:1788572585.646483:0:21190:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21190 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572585.646501:0:21190:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21190:1875453495420032:192.168.202.102@tcp:101:dd.500 00000100:00100000:2.0:1788572585.646637:0:21190:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d000 going to sleep for 11 seconds 00010000:00010000:0.0:1788572585.646743:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x11 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.646751:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117d6 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x11 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01eb1 expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.646764:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.646765:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.646767:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572585.646781:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495419776:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572585.659400:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495419776:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572585.660545:0:21190:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21190:1875453495420032:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572585.660589:0:21190:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 4/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660617:0:21190:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 4/0,1 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660626:0:21190:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 4/0,1 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660637:0:21190:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 4/0,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660647:0:21190:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 4/0,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660660:0:21190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 5/1,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660667:0:21190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 5/1,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660674:0:21190:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572585.660685:0:21190:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff93a24940ad80 x1875453495420032/t17179869193(17179869193) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000002:00100000:2.0:1788572585.660696:0:21190:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940ad80 x1875453495420032/t17179869193(17179869193) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000002:00010000:2.0:1788572585.660932:0:21190:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572585.660945:0:21190:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.660957:0:21190:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/2 (0 0) 00000080:00200000:2.0:1788572585.660963:0:21190:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200001b71:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572585.663603:0:21190:0:(namei.c:1628:ll_create_it()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=open|creat 00000080:00200000:2.0:1788572585.663641:0:21190:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00200000:2.0:1788572585.664618:0:21190:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 0 00000080:00200000:2.0:1788572585.664642:0:21190:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572585.664647:0:21190:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00010000:2.0:1788572585.664650:0:21190:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb117dd 00000080:00010000:2.0:1788572585.664658:0:21190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00000400:2.0:1788572585.664896:0:21190:0:(pcc.c:3550:pcc_file_reset_projid()) /mnt/pcc.d1f.sanity-pcc/d1f.sanity-pcc: cache fs project quota off, disabling: rc = -95 00000080:00200008:2.0:1788572585.679931:0:21190:0:(file.c:1090:ll_file_open()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 101101 00000002:00100000:2.0:1788572585.679982:0:21190:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940ad80 x1875453495420032/t17179869193(17179869193) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000080:00010000:2.0:1788572585.680024:0:21190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117dd from it ffff93a242ca0420 00010000:00010000:2.0:1788572585.680034:0:21190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.680048:0:21190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572585.681154:0:21190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572585.681327:0:21190:0:(file.c:2650:do_file_write_iter()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0], ppos: 0, count: 1024 00000080:00200008:2.0:1788572585.681515:0:21190:0:(file.c:479:ll_file_release()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 100001 00000002:00100000:2.0:1788572585.681575:0:21190:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940ad80 x1875453495420032/t17179869193(17179869193) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572601 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:100 00000020:00100000:2.0:1788572585.681595:0:21190:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572585.681600:0:21190:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940aa00 x1875453495420288/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:2.0:1788572585.681623:0:21190:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21190:1875453495420288:192.168.202.102@tcp:35:dd.500 00000100:00100000:2.0:1788572585.681717:0:21190:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572585.686851:0:21190:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21190:1875453495420288:192.168.202.102@tcp:35:dd.500 00000002:00100000:2.0:1788572585.686889:0:21190:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940ad80 x1875453495420032/t17179869193(17179869193) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:100 00000100:00100000:1.0:1788572585.815469:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495420544:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572585.815568:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96693705-5e29-4404-974e-e904d01a815e:2353:1875453495420800:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572585.815602:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96693705-5e29-4404-974e-e904d01a815e:2353:1875453495421056:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572585.815630:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1788572585.815637:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495420416:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572585.815660:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495420672:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572585.815679:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96693705-5e29-4404-974e-e904d01a815e:2353:1875453495420928:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572585.826871:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495420416:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572585.826911:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495420672:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572585.826926:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96693705-5e29-4404-974e-e904d01a815e:2353:1875453495420928:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572585.826943:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495420544:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572585.826979:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96693705-5e29-4404-974e-e904d01a815e:2353:1875453495420800:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572585.826995:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:96693705-5e29-4404-974e-e904d01a815e:2353:1875453495421056:192.168.202.102@tcp:400:kworker.0 00000080:00200000:2.0:1788572585.844027:0:21200:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572585.846407:0:21200:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572585.846440:0:21200:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572585.846475:0:21200:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788572585.846482:0:21200:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572585.846492:0:21200:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572585.846496:0:21200:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:0.0:1788572585.846504:0:21200:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f9fbf8 00010000:00010000:0.0:1788572585.846508:0:21200:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572585.846517:0:21200:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788572585.846536:0:21200:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572585.846563:0:21200:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), file ffff93a250dfb500, open_flags 18800, mode 0 opened 0 00000080:00200000:0.0:1788572585.846570:0:21200:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=open 00000002:00010000:0.0:1788572585.846588:0:21200:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:0.0:1788572585.846644:0:21200:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.846651:0:21200:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572585.846663:0:21200:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572585.846677:0:21200:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d0380 x1875453495421312/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572585.846711:0:21200:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572585.846736:0:21200:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21200:1875453495421312:192.168.202.102@tcp:101:lfs.0 00000100:00100000:0.0:1788572585.847123:0:21200:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2429cb3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572585.856457:0:21200:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21200:1875453495421312:192.168.202.102@tcp:101:lfs.0 00010000:00010000:0.0:1788572585.856482:0:21200:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856497:0:21200:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856505:0:21200:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856511:0:21200:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856521:0:21200:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856529:0:21200:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742ade00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:0.0:1788572585.856533:0:21200:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856539:0:21200:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856546:0:21200:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856568:0:21200:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572585.856575:0:21200:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.856582:0:21200:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117e4 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21200 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572585.856603:0:21200:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a2460d0380 x1875453495421312/t17179869195(17179869195) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572585.856616:0:21200:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a2460d0380 x1875453495421312/t17179869195(17179869195) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572585.856628:0:21200:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572585.856650:0:21200:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200000:0.0:1788572585.856654:0:21200:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x1:0x0](ffff93a25110ec08): rc = 0 00000080:00010000:0.0:1788572585.856657:0:21200:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x0 00000080:00200008:0.0:1788572585.856682:0:21200:0:(file.c:1090:ll_file_open()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00100000:0.0:1788572585.856715:0:21200:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a2460d0380 x1875453495421312/t17179869195(17179869195) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572585.856761:0:21200:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 44 00000080:00200080:0.0:1788572585.856820:0:21200:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) cmd=901066fc arg=7ffd9a4039d0 00000080:00200008:0.0:1788572585.856859:0:21200:0:(file.c:479:ll_file_release()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00100000:0.0:1788572585.857126:0:21200:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a2460d0380 x1875453495421312/t17179869195(17179869195) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572585.857138:0:21200:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572585.857143:0:21200:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d3800 x1875453495421568/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:0.0:1788572585.857299:0:21200:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21200:1875453495421568:192.168.202.102@tcp:35:lfs.0 00000100:00100000:0.0:1788572585.857375:0:21200:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2429cb3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572585.867837:0:21200:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21200:1875453495421568:192.168.202.102@tcp:35:lfs.0 00000002:00100000:0.0:1788572585.867877:0:21200:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a2460d0380 x1875453495421312/t17179869195(17179869195) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572601 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572586.017927:0:21206:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572586.017953:0:21206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572586.017972:0:21206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788572586.017989:0:21206:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788572586.017996:0:21206:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:0.0:1788572586.018003:0:21206:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572586.018007:0:21206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:0.0:1788572586.018014:0:21206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f2fbf8 00010000:00010000:0.0:1788572586.018018:0:21206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788572586.018028:0:21206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:0.0:1788572586.018056:0:21206:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572586.018071:0:21206:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), flags=25 00000080:00200000:0.0:1788572586.018080:0:21206:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d1f.sanity-pcc, dir=[0x200000007:0x1:0x0](ffff93a251173a88), intent=0 00000002:00010000:0.0:1788572586.018092:0:21206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572586.018135:0:21206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21206 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.018142:0:21206:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21206 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.018152:0:21206:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21206 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572586.018178:0:21206:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21206:1875453495421696:192.168.202.102@tcp:101:dd.500 00000100:00100000:1.0:1788572586.019135:0:21206:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2429cb3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.050481:0:21206:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21206:1875453495421696:192.168.202.102@tcp:101:dd.500 00010000:00010000:3.0:1788572586.050511:0:21206:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050545:0:21206:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050553:0:21206:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050565:0:21206:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050576:0:21206:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050589:0:21206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050615:0:21206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050623:0:21206:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.050635:0:21206:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff93a2460d3800 x1875453495421696/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:0 00000002:00010000:3.0:1788572586.050647:0:21206:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050656:0:21206:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/1 (0 0) 00000080:00200000:3.0:1788572586.050662:0:21206:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1788572586.050684:0:21206:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00200000:3.0:1788572586.050687:0:21206:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x1:0x0](ffff93a25110ec08): rc = 0 00000080:00010000:3.0:1788572586.050692:0:21206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb117eb 00000080:00010000:3.0:1788572586.050705:0:21206:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572586.050708:0:21206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117eb from it ffffbaf6c1f2fac8 00010000:00010000:3.0:1788572586.050712:0:21206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050720:0:21206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572586.050762:0:21206:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:3.0:1788572586.050775:0:21206:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1f.sanity-pcc, flags=265 00000080:00200000:3.0:1788572586.050786:0:21206:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 44 00000080:00200008:3.0:1788572586.050804:0:21206:0:(file.c:1090:ll_file_open()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 100000 00000002:00010000:3.0:1788572586.050823:0:21206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x2:0x0]) in obj [0x200001b71:0x1:0x0], intent: open flags 0240100001 00010000:00010000:3.0:1788572586.050869:0:21206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21206 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.050876:0:21206:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21206 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572586.050886:0:21206:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572586.050893:0:21206:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c000 x1875453495421952/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:3.0:1788572586.050902:0:21206:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21206 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572586.050920:0:21206:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21206:1875453495421952:192.168.202.102@tcp:101:dd.500 00000100:00100000:3.0:1788572586.051110:0:21206:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.062324:0:21206:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21206:1875453495421952:192.168.202.102@tcp:101:dd.500 00010000:00010000:3.0:1788572586.062368:0:21206:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062384:0:21206:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062392:0:21206:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062402:0:21206:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062430:0:21206:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062443:0:21206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062450:0:21206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.062457:0:21206:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.062469:0:21206:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a25064c000 x1875453495421952/t17179869197(17179869197) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000002:00100000:3.0:1788572586.062482:0:21206:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064c000 x1875453495421952/t17179869197(17179869197) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000002:00010000:3.0:1788572586.062493:0:21206:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572586.062506:0:21206:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.063565:0:21206:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/2 (0 0) 00000080:00200000:3.0:1788572586.063575:0:21206:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200001b71:0x1:0x0] suppgids -1 500: rc 0 00000080:00200000:3.0:1788572586.063649:0:21206:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 0 00000080:00200000:3.0:1788572586.063676:0:21206:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788572586.063682:0:21206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb117f2 00000080:00010000:3.0:1788572586.063690:0:21206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117f2 from it ffffbaf6c1f2fbf0 00010000:00010000:3.0:1788572586.063694:0:21206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.063706:0:21206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1788572586.063737:0:21206:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a25064c000 x1875453495421952/t17179869197(17179869197) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000080:00200008:3.0:1788572586.063821:0:21206:0:(file.c:5214:ll_file_seek()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:3.0:1788572586.063829:0:21206:0:(file.c:5245:ll_file_seek()) COMPLETED file f1f.sanity-pcc:[0x200001b71:0x2:0x0], offset: 0, type: SEEK_CUR, rc = 0 00000080:00200008:3.0:1788572586.063936:0:21206:0:(file.c:2483:do_file_read_iter()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0], ppos: 0, count: 1024 00000080:00200008:3.0:1788572586.064008:0:21206:0:(file.c:479:ll_file_release()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 100000 00000002:00100000:2.0:1788572586.064135:0:21206:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a25064c000 x1875453495421952/t17179869197(17179869197) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:100 00000020:00100000:2.0:1788572586.064146:0:21206:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572586.064152:0:21206:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495422208/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:2.0:1788572586.064175:0:21206:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21206:1875453495422208:192.168.202.102@tcp:35:dd.500 00000100:00100000:2.0:1788572586.064287:0:21206:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.083191:0:21206:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21206:1875453495422208:192.168.202.102@tcp:35:dd.500 00000002:00100000:2.0:1788572586.083296:0:21206:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a25064c000 x1875453495421952/t17179869197(17179869197) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:100 00000080:00200000:3.0:1788572586.204576:0:21212:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572586.204604:0:21212:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.204622:0:21212:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572586.204637:0:21212:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572586.204645:0:21212:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572586.204653:0:21212:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572586.204657:0:21212:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572586.204664:0:21212:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f97c08 00010000:00010000:3.0:1788572586.204668:0:21212:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572586.204683:0:21212:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572586.204698:0:21212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572586.204706:0:21212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=25 00000080:00200000:3.0:1788572586.204712:0:21212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:3.0:1788572586.204716:0:21212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1f.sanity-pcc, flags=9 00000080:00200000:3.0:1788572586.204723:0:21212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 2 00000080:00200008:3.0:1788572586.204732:0:21212:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08) current size 0, valid attrs 8, mode ffff, uid -1591447252, gid -1, new size 256, atime 1788572586.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000 00010000:00010000:3.0:1788572586.204755:0:21212:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.204769:0:21212:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572586.204818:0:21212:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.204826:0:21212:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572586.204829:0:21212:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f2 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01f1a expref: -99 pid: 21206 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572586.204846:0:21212:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572586.204859:0:21212:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064d880 x1875453495422464/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572586.204886:0:21212:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job truncate:bf89ae46-999b-46d3-9a62-08dfe43909d0:21212:1875453495422464:192.168.202.102@tcp:36:truncate.500 00000100:00100000:3.0:1788572586.204980:0:21212:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a250bbb9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.211216:0:21212:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job truncate:bf89ae46-999b-46d3-9a62-08dfe43909d0:21212:1875453495422464:192.168.202.102@tcp:36:truncate.500 00000080:00200008:3.0:1788572586.211527:0:21212:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08) current size 0, valid attrs 10000168, mode ffff, uid -1591447252, gid -1, new size 256, atime 1788572586.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000, rc 0 00010000:00010000:2.0:1788572586.268719:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572586.327027:0:21218:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572586.327070:0:21218:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.327163:0:21218:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572586.327181:0:21218:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572586.327189:0:21218:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572586.327198:0:21218:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572586.327202:0:21218:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572586.327210:0:21218:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1f17c08 00010000:00010000:3.0:1788572586.327213:0:21218:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572586.327223:0:21218:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572586.327376:0:21218:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572586.327385:0:21218:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=25 00000080:00200000:3.0:1788572586.327390:0:21218:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:3.0:1788572586.327402:0:21218:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), flags=9 00000080:00200000:3.0:1788572586.327406:0:21218:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=getattr 00000002:00010000:3.0:1788572586.327424:0:21218:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572586.327469:0:21218:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21218 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.327477:0:21218:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21218 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.327488:0:21218:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21218 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572586.327997:0:21218:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job truncate:bf89ae46-999b-46d3-9a62-08dfe43909d0:21218:1875453495422592:192.168.202.102@tcp:101:truncate.500 00000100:00100000:3.0:1788572586.328261:0:21218:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24867c540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.340537:0:21218:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job truncate:bf89ae46-999b-46d3-9a62-08dfe43909d0:21218:1875453495422592:192.168.202.102@tcp:101:truncate.500 00010000:00010000:3.0:1788572586.340565:0:21218:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340583:0:21218:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340592:0:21218:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340605:0:21218:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340617:0:21218:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340629:0:21218:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340653:0:21218:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340661:0:21218:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.340673:0:21218:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064c000 x1875453495422592/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'truncate.500' uid:500 gid:500 projid:100 00000002:00010000:3.0:1788572586.340685:0:21218:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572586.340698:0:21218:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340709:0:21218:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/2 (0 0) 00000080:00200000:3.0:1788572586.340715:0:21218:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200001b71:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1788572586.340756:0:21218:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 256 00000080:00200000:3.0:1788572586.340760:0:21218:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x2:0x0](ffff93a251196c08): rc = 0 00000080:00200000:3.0:1788572586.340771:0:21218:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788572586.340776:0:21218:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb117f9 00000080:00010000:3.0:1788572586.340787:0:21218:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00010000:3.0:1788572586.340791:0:21218:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117f9 from it ffffbaf6c1f17bd0 00010000:00010000:3.0:1788572586.340795:0:21218:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.340804:0:21218:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572586.340847:0:21218:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 2 00000080:00200008:3.0:1788572586.340860:0:21218:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08) current size 256, valid attrs 8, mode ffff, uid -1591447252, gid -1, new size 2048, atime 1788572586.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000 00010000:00010000:3.0:1788572586.340882:0:21218:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.340914:0:21218:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572586.340956:0:21218:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.340962:0:21218:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572586.340965:0:21218:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb117f9 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01f44 expref: -99 pid: 21218 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572586.340982:0:21218:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572586.340989:0:21218:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064c000 x1875453495422848/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572586.341010:0:21218:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job truncate:bf89ae46-999b-46d3-9a62-08dfe43909d0:21218:1875453495422848:192.168.202.102@tcp:36:truncate.500 00000100:00100000:3.0:1788572586.341097:0:21218:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24867c540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.350307:0:21218:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job truncate:bf89ae46-999b-46d3-9a62-08dfe43909d0:21218:1875453495422848:192.168.202.102@tcp:36:truncate.500 00000080:00200008:3.0:1788572586.350393:0:21218:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08) current size 256, valid attrs 10000168, mode ffff, uid -1591447252, gid -1, new size 2048, atime 1788572586.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000, rc 0 00000080:00200000:1.0:1788572586.428463:0:21224:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:1.0:1788572586.428487:0:21224:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572586.428557:0:21224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572586.428570:0:21224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572586.428575:0:21224:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:1.0:1788572586.428582:0:21224:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:1.0:1788572586.428585:0:21224:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:1.0:1788572586.428591:0:21224:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fafbf8 00010000:00010000:1.0:1788572586.428594:0:21224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572586.428610:0:21224:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572586.428622:0:21224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572586.428628:0:21224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=25 00000080:00200000:1.0:1788572586.428632:0:21224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:1.0:1788572586.428639:0:21224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 3 00000080:00200000:1.0:1788572586.428642:0:21224:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), file ffff93a2429e2900, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1788572586.428649:0:21224:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=open|creat 00000002:00010000:1.0:1788572586.428667:0:21224:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1f.sanity-pcc,[0x200001b71:0x3:0x0]) in obj [0x200001b71:0x1:0x0], intent: open|creat flags 010000000101102 00010000:00010000:1.0:1788572586.428708:0:21224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21224 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.428714:0:21224:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21224 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572586.428722:0:21224:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572586.428733:0:21224:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a258da1c00 x1875453495423104/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:1.0:1788572586.428740:0:21224:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21224 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572586.428757:0:21224:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21224:1875453495423104:192.168.202.102@tcp:101:dd.500 00000100:00100000:1.0:1788572586.428882:0:21224:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a253e82e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.432680:0:21224:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21224:1875453495423104:192.168.202.102@tcp:101:dd.500 00010000:00010000:2.0:1788572586.432709:0:21224:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 4/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432727:0:21224:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 4/0,1 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432735:0:21224:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 4/0,1 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432747:0:21224:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 4/0,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432756:0:21224:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 4/0,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432769:0:21224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 5/1,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432777:0:21224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 5/1,1 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432783:0:21224:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572586.432809:0:21224:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x2b, status=0 req@ffff93a258da1c00 x1875453495423104/t17179869203(17179869203) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000002:00100000:2.0:1788572586.432821:0:21224:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da1c00 x1875453495423104/t17179869203(17179869203) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000002:00010000:2.0:1788572586.432832:0:21224:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572586.432843:0:21224:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432855:0:21224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572586.432865:0:21224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572586.432872:0:21224:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432879:0:21224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432885:0:21224:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.432901:0:21224:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572586.432912:0:21224:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200001b71:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572586.433135:0:21224:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 2048 00000080:00200000:2.0:1788572586.433139:0:21224:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x2:0x0](ffff93a251196c08): rc = 0 00000080:00200000:2.0:1788572586.433149:0:21224:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788572586.433154:0:21224:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb117dd 00000080:00010000:2.0:1788572586.433163:0:21224:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00200008:2.0:1788572586.433223:0:21224:0:(file.c:1090:ll_file_open()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 101101 00010000:00010000:0.0:1788572586.433365:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572586.433372:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433381:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742c7400 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:0.0:1788572586.433386:0:20561:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x9 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433394:0:20561:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x9 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433404:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 3/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x9 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433422:0:20561:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572586.433458:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x9 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433471:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572586.433870:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x9 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433878:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7400/0x6d478a807bb11800 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x9 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01f67 expref: -99 pid: 21224 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.433892:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572586.433893:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572586.433895:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572586.433905:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572586.433923:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495423232:192.168.202.102@tcp:103:ldlm_bl.0 00000002:00100000:2.0:1788572586.435386:0:21224:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a258da1c00 x1875453495423104/t17179869203(17179869203) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.500' uid:500 gid:500 projid:100 00000080:00010000:2.0:1788572586.435424:0:21224:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117dd from it ffff93a247a25900 00010000:00010000:2.0:1788572586.435432:0:21224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572586.435443:0:21224:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572586.435480:0:21224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 42 00000080:00200008:2.0:1788572586.435495:0:21224:0:(llite_lib.c:2427:ll_setattr_raw()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08) current size 2048, valid attrs a068, mode 0, uid 2, gid 0, new size 0, atime 1788572586.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000 00000020:00100000:2.0:1788572586.435544:0:21224:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572586.435552:0:21224:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495423488/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572586.435574:0:21224:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21224:1875453495423488:192.168.202.102@tcp:36:dd.500 00000100:00100000:2.0:1788572586.435639:0:21224:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517dd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572586.439441:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495423232:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572586.439475:0:21224:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21224:1875453495423488:192.168.202.102@tcp:36:dd.500 00000080:00200008:2.0:1788572586.439634:0:21224:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08) current size 2048, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788572586.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000, rc 0 00000080:00200008:2.0:1788572586.439729:0:21224:0:(file.c:2650:do_file_write_iter()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0], ppos: 0, count: 256 00000080:00200008:2.0:1788572586.439861:0:21224:0:(file.c:479:ll_file_release()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 100001 00000002:00100000:2.0:1788572586.439908:0:21224:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a258da1c00 x1875453495423104/t17179869203(17179869203) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:100 00000020:00100000:2.0:1788572586.439917:0:21224:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572586.439921:0:21224:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409f80 x1875453495423744/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:2.0:1788572586.439937:0:21224:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21224:1875453495423744:192.168.202.102@tcp:35:dd.500 00000100:00100000:2.0:1788572586.440008:0:21224:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.442189:0:21224:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job dd:bf89ae46-999b-46d3-9a62-08dfe43909d0:21224:1875453495423744:192.168.202.102@tcp:35:dd.500 00000002:00100000:2.0:1788572586.442223:0:21224:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a258da1c00 x1875453495423104/t17179869203(17179869203) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.500' uid:500 gid:500 projid:100 00000080:00200000:0.0:1788572586.508223:0:21234:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:0.0:1788572586.508401:0:21234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572586.508420:0:21234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572586.508436:0:21234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572586.508443:0:21234:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572586.508451:0:21234:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:0.0:1788572586.508454:0:21234:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:0.0:1788572586.508461:0:21234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fd7bf8 00010000:00010000:0.0:1788572586.508464:0:21234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572586.508481:0:21234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788572586.508496:0:21234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572586.508505:0:21234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=267 00000080:00200000:0.0:1788572586.508513:0:21234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 44 00000080:00200008:0.0:1788572586.508520:0:21234:0:(file.c:1090:ll_file_open()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00010000:0.0:1788572586.508536:0:21234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788572586.508591:0:21234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.508597:0:21234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572586.508607:0:21234:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572586.508622:0:21234:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d2d80 x1875453495424000/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:0.0:1788572586.508632:0:21234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572586.508658:0:21234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2d80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21234:1875453495424000:192.168.202.102@tcp:101:lfs.0 00000100:00100000:0.0:1788572586.508904:0:21234:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2581a8cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572586.512075:0:21234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2d80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21234:1875453495424000:192.168.202.102@tcp:101:lfs.0 00010000:00010000:0.0:1788572586.512098:0:21234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512109:0:21234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512124:0:21234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512128:0:21234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512134:0:21234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512137:0:21234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a2742adc00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:0.0:1788572586.512140:0:21234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512145:0:21234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512150:0:21234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512154:0:21234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572586.512160:0:21234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.512166:0:21234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742adc00/0x6d478a807bb11807 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572586.512184:0:21234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a2460d2d80 x1875453495424000/t17179869207(17179869207) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:100 00000002:00100000:0.0:1788572586.512196:0:21234:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a2460d2d80 x1875453495424000/t17179869207(17179869207) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:100 00000080:00200000:0.0:1788572586.512214:0:21234:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572586.512261:0:21234:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000002:00100000:0.0:1788572586.512275:0:21234:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a2460d2d80 x1875453495424000/t17179869207(17179869207) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:100 00000080:00200080:0.0:1788572586.512355:0:21234:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) cmd=901066fc arg=7fff3d7ad0c0 00000080:00200008:0.0:1788572586.512386:0:21234:0:(file.c:479:ll_file_release()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00100000:0.0:1788572586.512423:0:21234:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a2460d2d80 x1875453495424000/t17179869207(17179869207) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572602 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:100 00000020:00100000:0.0:1788572586.512430:0:21234:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572586.512433:0:21234:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d0e00 x1875453495424256/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:0.0:1788572586.512447:0:21234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0e00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21234:1875453495424256:192.168.202.102@tcp:35:lfs.0 00000100:00100000:0.0:1788572586.512496:0:21234:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a2581a8cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572586.515664:0:21234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0e00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21234:1875453495424256:192.168.202.102@tcp:35:lfs.0 00000002:00100000:0.0:1788572586.515750:0:21234:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a2460d2d80 x1875453495424000/t17179869207(17179869207) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:100 00000080:00200000:2.0:1788572586.540713:0:21237:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572586.540746:0:21237:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.540766:0:21237:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788572586.540785:0:21237:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788572586.540793:0:21237:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.540801:0:21237:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572586.540805:0:21237:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:2.0:1788572586.540813:0:21237:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fefb98 00010000:00010000:2.0:1788572586.540825:0:21237:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788572586.540832:0:21237:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572586.540847:0:21237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572586.540853:0:21237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=24 00000080:00200000:2.0:1788572586.540857:0:21237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1788572586.540861:0:21237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1f.sanity-pcc, flags=8 00000080:00200008:3.0:1788572587.047920:0:21271:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff93a251173a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572587.047940:0:21271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572587.047961:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572587.047981:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572587.047997:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788572587.048018:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572587.048027:0:21271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572587.048031:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572587.048039:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fefcb8 00010000:00010000:3.0:1788572587.048043:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1788572587.048053:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1788572587.048082:0:21271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572587.048089:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572587.048096:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1788572587.048104:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1788572587.048111:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572587.048114:0:21271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572587.048124:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572587.048128:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fefb58 00010000:00010000:3.0:1788572587.048132:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1788572587.048139:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1788572587.048149:0:21271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572587.048156:0:21271:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=8 00000080:00200008:3.0:1788572587.048163:0:21271:0:(file.c:6296:ll_getattr_dentry()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572587.048167:0:21271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08),name=d1f.sanity-pcc 00000002:00010000:3.0:1788572587.048178:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572587.048184:0:21271:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 30 res 8589941617/1 (0 0) 00010000:00010000:3.0:1788572587.048427:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.048437:0:21271:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.048449:0:21271:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572587.048476:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495424384:192.168.202.102@tcp:101:lfs.500 00000100:00100000:3.0:1788572587.048640:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24867c540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572587.055541:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495424384:192.168.202.102@tcp:101:lfs.500 00010000:00010000:3.0:1788572587.055570:0:21271:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055607:0:21271:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055615:0:21271:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055626:0:21271:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055638:0:21271:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055651:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055660:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055667:0:21271:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572587.055679:0:21271:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064ea00 x1875453495424384/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:100 00000002:00010000:3.0:1788572587.055691:0:21271:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055699:0:21271:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/1 (0 0) 00000080:00200000:3.0:1788572587.055710:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1788572587.055730:0:21271:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00010000:3.0:1788572587.055735:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb1180e 00000080:00010000:3.0:1788572587.055742:0:21271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) 00000080:00010000:3.0:1788572587.055746:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1180e from it ffffbaf6c1fefcb8 00010000:00010000:3.0:1788572587.055749:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055757:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572587.055810:0:21271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:3.0:1788572587.055821:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572587.055830:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788572587.055851:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1788572587.055857:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:3.0:1788572587.055861:0:21271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:3.0:1788572587.055864:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:3.0:1788572587.055867:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fefb58 00010000:00010000:3.0:1788572587.055871:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788572587.055878:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:3.0:1788572587.055897:0:21271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572587.055903:0:21271:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=24 00000080:00200000:3.0:1788572587.055907:0:21271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:3.0:1788572587.055912:0:21271:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1f.sanity-pcc, flags=8 00000080:00200008:3.0:1788572587.055928:0:21271:0:(file.c:6296:ll_getattr_dentry()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572587.055933:0:21271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08),name=f1f.sanity-pcc 00000002:00010000:3.0:1788572587.055943:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x2:0x0]) in obj [0x200001b71:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572587.055948:0:21271:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 30 res 8589941617/2 (0 0) 00010000:00010000:3.0:1788572587.055982:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055988:0:21271:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.055996:0:21271:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572587.056025:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495424512:192.168.202.102@tcp:101:lfs.500 00000100:00100000:3.0:1788572587.056154:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24867c540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572587.062992:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495424512:192.168.202.102@tcp:101:lfs.500 00010000:00010000:3.0:1788572587.063037:0:21271:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063446:0:21271:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063455:0:21271:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063600:0:21271:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063613:0:21271:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063627:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063637:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.063644:0:21271:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572587.063659:0:21271:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a25064ea00 x1875453495424512/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:100 00000002:00010000:3.0:1788572587.063670:0:21271:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572587.063792:0:21271:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.063805:0:21271:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/2 (0 0) 00000080:00200000:2.0:1788572587.063812:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200001b71:0x1:0x0] suppgids -1 500: rc 0 00000080:00200000:2.0:1788572587.063856:0:21271:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 0 00000080:00200000:2.0:1788572587.063871:0:21271:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788572587.063876:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb11815 00000080:00010000:2.0:1788572587.063882:0:21271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00010000:2.0:1788572587.063885:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11815 from it ffffbaf6c1fefcb8 00010000:00010000:2.0:1788572587.063889:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.063899:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.064025:0:21271:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572587.064039:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572587.064116:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788572587.064135:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788572587.064142:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids -1 -1: rc 1 00000080:00010000:2.0:1788572587.064148:0:21271:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572587.064151:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:2.0:1788572587.064155:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fefbf8 00010000:00010000:2.0:1788572587.064158:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788572587.064167:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788572587.064177:0:21271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572587.064186:0:21271:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=25 00000080:00200000:2.0:1788572587.064191:0:21271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1788572587.064195:0:21271:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f1f.sanity-pcc, flags=265 00000080:00200000:2.0:1788572587.064199:0:21271:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 46 00000080:00200008:2.0:1788572587.064207:0:21271:0:(file.c:1090:ll_file_open()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 154402 00000002:00010000:2.0:1788572587.064222:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x2:0x0]) in obj [0x200001b71:0x1:0x0], intent: open flags 0100240114403 00010000:00010000:2.0:1788572587.067386:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.067407:0:21271:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572587.067419:0:21271:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572587.067445:0:21271:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409f80 x1875453495424768/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:2.0:1788572587.067454:0:21271:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.067592:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495424768:192.168.202.102@tcp:101:lfs.500 00000100:00100000:2.0:1788572587.067786:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.079686:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495424768:192.168.202.102@tcp:101:lfs.500 00010000:00010000:2.0:1788572587.079716:0:21271:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079732:0:21271:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079740:0:21271:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079750:0:21271:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079762:0:21271:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079775:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079783:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079810:0:21271:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.079823:0:21271:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a249409f80 x1875453495424768/t17179869209(17179869209) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:100 00000002:00100000:2.0:1788572587.079847:0:21271:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409f80 x1875453495424768/t17179869209(17179869209) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:100 00000002:00010000:2.0:1788572587.079862:0:21271:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572587.079876:0:21271:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079890:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.079904:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.079913:0:21271:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079919:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079926:0:21271:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.079959:0:21271:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572587.079967:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200001b71:0x1:0x0] suppgids -1 500: rc 0 00000080:00200000:2.0:1788572587.080018:0:21271:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 0 00000080:00200000:2.0:1788572587.080032:0:21271:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788572587.080037:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb11815 00000080:00010000:2.0:1788572587.080053:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11815 from it ffffbaf6c1fefbf0 00010000:00010000:2.0:1788572587.080057:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.080067:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1788572587.080091:0:21271:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409f80 x1875453495424768/t17179869209(17179869209) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.500' uid:500 gid:500 projid:100 00000080:00200080:2.0:1788572587.080182:0:21271:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x200001b71:0x2:0x0](0000000000000000) cmd=c00466fc arg=7fffaa1e1fdc 00800000:00000080:2.0:1788572587.080201:0:21271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000002:00000080:2.0:1788572587.080208:0:21271:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00010000:00010000:2.0:1788572587.080441:0:21271:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 3/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572587.080456:0:21271:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b71:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572587.080470:0:21271:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572587.080478:0:21271:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b71:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572587.080484:0:21271:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572587.080501:0:21271:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b71:0x2:0x0]: losing layout lock 00010000:00010000:3.0:1788572587.080600:0:20560:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572587.080607:0:20560:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.080615:0:20560:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216a00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:3.0:1788572587.080619:0:20560:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.080633:0:20560:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.080640:0:20560:0:(ldlm_request.c:1774:ldlm_cli_cancel()) ### lock is already being destroyed 00010000:00010000:3.0:1788572587.080643:0:20560:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.080651:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572587.080653:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572587.080655:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.080697:0:21271:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.080705:0:21271:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572587.080711:0:21271:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572587.080716:0:21271:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:2.0:1788572587.080735:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495425024:192.168.202.102@tcp:103:lfs.0 00000100:00100000:2.0:1788572587.080893:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.090025:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495425024:192.168.202.102@tcp:103:lfs.0 00010000:00010000:2.0:1788572587.090066:0:21271:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1181c lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01fad expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.090087:0:21271:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9800/0x6d478a807bb11815 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01f9f expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572587.090098:0:21271:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb117dd lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01ecd expref: -99 pid: 21190 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788572587.090125:0:21271:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 58, max 7 00000100:00100000:2.0:1788572587.090134:0:21271:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409c00 x1875453495425152/t0(0) o58->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572587.090180:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495425152:192.168.202.102@tcp:58:lfs.0 00000100:00100000:2.0:1788572587.090320:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.096810:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495425152:192.168.202.102@tcp:58:lfs.0 00000080:00010000:2.0:1788572587.096864:0:21271:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200001b71:0x2:0x0](ffff93a251196c08) 00000002:00010000:2.0:1788572587.096877:0:21271:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x2:0x0]) in obj [0x200001b71:0x2:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572587.096930:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.096940:0:21271:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.096950:0:21271:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21271 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.096970:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495425280:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572587.097074:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.099541:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495425280:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572587.099581:0:21271:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.099599:0:21271:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.286486:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572587.441331:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572587.445202:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518464384 00000100:00100000:1.0:1788572587.445554:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518464384 00000100:00100000:1.0:1788572587.445691:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 3 00000100:00100000:1.0:1788572587.445702:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23689:x1875453518464384:12345-192.168.202.102@tcp:107:hsm_cdtr.0 00000100:00100000:1.0:1788572587.445844:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23689:x1875453518464384:12345-192.168.202.102@tcp:107:hsm_cdtr.0 Request processed in 143us (4523us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572587.445851:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 3 00000080:00200080:2.0:1788572587.447786:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c0086696 arg=7fcba4000b60 00000100:00100000:2.0:1788572587.447853:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425408:192.168.202.102@tcp:900:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.447940:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.462286:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425408:192.168.202.102@tcp:900:lhsmtool_posix.0 00800000:00000080:2.0:1788572587.462322:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c0086696 len=4128 karg=ffff93a2491ce000 uarg=0000000000000000 00000002:00000080:2.0:1788572587.462333:0:21272:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=c0086696 len=4128 karg=ffff93a2491ce000 uarg=0000000000000000 00000002:00000080:2.0:1788572587.462337:0:21272:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x200001b71:0x2:0x0] from 18446744073709551615 #0 00000100:00100000:2.0:1788572587.462370:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425536:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.462451:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.465353:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425536:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572587.465381:0:21272:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1f.sanity-pcc/f1f.sanity-pcc 00000080:00200080:2.0:1788572587.465538:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c01066f8 arg=c3e9e8 00800000:00000080:2.0:1788572587.465550:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c01066f8 len=16 karg=ffffbaf6c1fafcf0 uarg=ffffbaf6c1fafcc0 00000080:00200080:2.0:1788572587.465977:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110c2c8) cmd=c1406916 arg=7fcba4000ce0 00000100:00100000:2.0:1788572587.466050:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425664:192.168.202.102@tcp:34:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.466137:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.471282:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425664:192.168.202.102@tcp:34:lhsmtool_posix.0 00000080:00200000:2.0:1788572587.471362:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110c2c8), inode mode 4040 mask 1 00000080:00200000:2.0:1788572587.471380:0:21272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=0x200001b71:0x2:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110c2c8), file ffff93a250dfa780, open_flags 68800, mode 0 opened 0 00000080:00200000:2.0:1788572587.471387:0:21272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=0x200001b71:0x2:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110c2c8), intent=open 00000002:00010000:2.0:1788572587.471413:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x2:0x0]) in obj [0x200000002:0x2:0x0], intent: open flags 041504001 00010000:00010000:2.0:1788572587.471484:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.471494:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572587.471518:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572587.471527:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409500 x1875453495425920/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572587.471536:0:21272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.471556:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425920:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.471721:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.478865:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495425920:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.478896:0:21272:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478910:0:21272:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478918:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478925:0:21272:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 3/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478935:0:21272:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478941:0:21272:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216a00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572587.478946:0:21272:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478972:0:21272:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 4/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478980:0:21272:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 5/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.478999:0:21272:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572587.479009:0:21272:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 3/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.479016:0:21272:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb1182a lrc: 0/0,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.479046:0:21272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a249409500 x1875453495425920/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572587.479061:0:21272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409500 x1875453495425920/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572587.479076:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572587.479120:0:21272:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x2:0x0], updating i_size 0 00000080:00200000:2.0:1788572587.479124:0:21272:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x2:0x0](ffff93a251196c08): rc = 0 00000080:00010000:2.0:1788572587.479128:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x0 00000080:00200008:2.0:1788572587.479150:0:21272:0:(file.c:1090:ll_file_open()) START file 0x200001b71:0x2:0x0:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 1504000 00000002:00100000:2.0:1788572587.479169:0:21272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409500 x1875453495425920/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572587.479293:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 44 00000080:00200080:2.0:1788572587.479327:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c0086696 arg=7fcba4000ce0 00800000:00000080:2.0:1788572587.479341:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c0086696 len=4128 karg=ffff93a2491ce000 uarg=0000000000000000 00000002:00000080:2.0:1788572587.479364:0:21272:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=c0086696 len=4128 karg=ffff93a2491ce000 uarg=0000000000000000 00000002:00000080:2.0:1788572587.479368:0:21272:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x200001b71:0x2:0x0] from 18446744073709551615 #0 00000100:00100000:2.0:1788572587.479401:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426048:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.480581:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.497791:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426048:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572587.497822:0:21272:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1f.sanity-pcc/f1f.sanity-pcc 00000080:00200000:2.0:1788572587.497944:0:21272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572587.497965:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572587.498018:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788572587.498043:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788572587.498053:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572587.498060:0:21272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572587.498064:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:2.0:1788572587.498071:0:21272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fafbf8 00010000:00010000:2.0:1788572587.498075:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788572587.498083:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572587.498099:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572587.498108:0:21272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=24 00000080:00200000:2.0:1788572587.498128:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1788572587.498144:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1788572587.498148:0:21272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:54A7F4B8:fd=09, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), file ffff93a250dfb200, open_flags 280c2, mode 8180 opened 0 00000080:00200000:2.0:1788572587.498155:0:21272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:54A7F4B8:fd=09, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=open|creat 00000002:00010000:2.0:1788572587.498185:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:54A7F4B8:fd=09,[0x200001b71:0x4:0x0]) in obj [0x200001b71:0x1:0x0], intent: open|creat flags 0500303 00010000:00010000:2.0:1788572587.498201:0:21272:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.498248:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742ade00/0x6d478a807bb117eb lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01f0c expref: -99 pid: 21206 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572587.498296:0:21272:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.498303:0:21272:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572587.498306:0:21272:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a253be9600/0x6d478a807bb1180e lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c01f91 expref: -99 pid: 21271 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.498335:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.498342:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572587.498353:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572587.498361:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940a680 x1875453495426304/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 680/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:2.0:1788572587.498370:0:21272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 3/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.498397:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426304:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.498758:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.505580:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426304:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.505614:0:21272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 4/0,1 mode: --/CW res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505635:0:21272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 4/0,1 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505643:0:21272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 4/0,1 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505654:0:21272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 4/0,1 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505666:0:21272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 4/0,1 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505678:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 5/1,1 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505686:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 5/1,1 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505694:0:21272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.505728:0:21272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff93a24940a680 x1875453495426304/t17179869211(17179869211) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 680/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000002:00100000:2.0:1788572587.505754:0:21272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940a680 x1875453495426304/t17179869211(17179869211) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 680/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572587.505778:0:21272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572587.505792:0:21272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.505801:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/4 (0 0) 00000080:00200000:2.0:1788572587.505808:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572587.505820:0:21272:0:(namei.c:1628:ll_create_it()) VFS Op:name=. :VOLATILE:0000:54A7F4B8:fd=09, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=open|creat 00000080:00200000:2.0:1788572587.505852:0:21272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200001b71:0x4:0x0](ffff93a251194b08) 00000080:00200000:2.0:1788572587.505928:0:21272:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x4:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572587.505933:0:21272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) 00000080:00010000:2.0:1788572587.505937:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) for lock 0x6d478a807bb11831 00000080:00010000:2.0:1788572587.505946:0:21272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) 00000080:00200008:2.0:1788572587.505959:0:21272:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08), flags 500302 00000002:00100000:2.0:1788572587.505978:0:21272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940a680 x1875453495426304/t17179869211(17179869211) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572603 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000080:00010000:2.0:1788572587.506022:0:21272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11831 from it ffff93a242ca0f00 00010000:00010000:2.0:1788572587.506038:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.506059:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.506084:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x4:0x0](ffff93a251194b08), inode mode 8180 mask 40 00000080:00200000:2.0:1788572587.506129:0:21272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572587.506142:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572587.506153:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572587.506164:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572587.506170:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572587.506175:0:21272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572587.506178:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:2.0:1788572587.506181:0:21272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fafc10 00010000:00010000:2.0:1788572587.506185:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572587.506192:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572587.506200:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572587.506206:0:21272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=24 00000080:00200000:2.0:1788572587.506211:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:2.0:1788572587.506263:0:21272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=. :VOLATILE:0000:54A7F4B8:fd=09, flags=0 00000080:00200000:2.0:1788572587.506275:0:21272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 3 00000080:00200000:2.0:1788572587.506280:0:21272:0:(namei.c:2351:ll_unlink()) VFS Op:name=. :VOLATILE:0000:54A7F4B8:fd=09, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08) 00010000:00010000:2.0:1788572587.506307:0:21272:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.506323:0:21272:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b71:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1788572587.506357:0:21272:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.506365:0:21272:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572587.506368:0:21272:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11831 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c01ffa expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572587.506394:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572587.506402:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408380 x1875453495426560/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572587.506423:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426560:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.506513:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.522167:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426560:192.168.202.102@tcp:36:lhsmtool_posix.0 00000002:00010000:2.0:1788572587.522306:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200001b71:0x4:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572587.522366:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 3/1,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.522377:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 3/1,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.522387:0:21272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 3/1,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.522426:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426688:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.522521:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.526351:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426688:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.526378:0:21272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.526392:0:21272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.526403:0:21272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.526416:0:21272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff93a249408380 x1875453495426688/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 568/1072 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572587.526427:0:21272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.526439:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/4 (0 0) 00000080:00200000:2.0:1788572587.526445:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x200001b71:0x4:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572587.526476:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) for lock 0x6d478a807bb11838 00000080:00010000:2.0:1788572587.526484:0:21272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11838 from it ffffbaf6c1fafc18 00010000:00010000:2.0:1788572587.526489:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.526499:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572587.526546:0:21272:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08) current size 0, valid attrs 1846, mode 93a2, uid 500, gid 500, new size 0, atime 1788572587.000000000, mtime 1788572587.000000000, ctime 1788572587.000000000 00000020:00100000:2.0:1788572587.526587:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572587.526594:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408380 x1875453495426944/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572587.526614:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426944:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.526692:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.540386:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495426944:192.168.202.102@tcp:36:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.540443:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 31 res 8589941617/4 (0 0) 00000080:00010000:2.0:1788572587.540463:0:21272:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200001b71:0x4:0x0](ffff93a251194b08) 00000002:00010000:2.0:1788572587.540473:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x4:0x0]) in obj [0x200001b71:0x4:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572587.540522:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.540531:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.540542:0:21272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.540562:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427072:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.540651:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.552816:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427072:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.552849:0:21272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 4/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.552880:0:21272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 4/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.552898:0:21272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a24940b100 x1875453495427072/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572587.552910:0:21272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572587.552935:0:21272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.552949:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 16 res 8589941617/4 (0 0) 00000080:00200000:2.0:1788572587.552954:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200001b71:0x4:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572587.552973:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) for lock 0x6d478a807bb1183f 00000080:00010000:2.0:1788572587.552991:0:21272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1183f from it ffffbaf6c1fafa68 00010000:00010000:2.0:1788572587.552996:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.553007:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.553641:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.553658:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572587.553682:0:21272:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200001b71:0x4:0x0](000000002415dad6) being reconfigured ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.553717:0:21272:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x4:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788572587.553724:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.553733:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572587.553753:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572587.553883:0:21272:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788572587.554125:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495427200:192.168.202.102@tcp:2:lhsmtool_posix.0 00000100:00100000:3.0:1788572587.556778:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495427200:192.168.202.102@tcp:2:lhsmtool_posix.0 00000080:00200000:2.0:1788572587.557005:0:21272:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572587.557022:0:21272:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08) current size 0, valid attrs 10001846, mode 93a2, uid 500, gid 500, new size 0, atime 1788572587.000000000, mtime 1788572587.000000000, ctime 1788572587.000000000, rc 0 00000080:00200008:2.0:1788572587.557140:0:21272:0:(file.c:479:ll_file_release()) START file 0x200001b71:0x2:0x0:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 1504000 00000002:00100000:2.0:1788572587.557346:0:21272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249409500 x1875453495425920/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572587.557467:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572587.557474:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940bb80 x1875453495427456/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:2.0:1788572587.557497:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427456:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.557583:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.562142:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427456:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:2.0:1788572587.562382:0:21272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249409500 x1875453495425920/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572587.562441:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=400866d6 arg=7fcba4000b78 00800000:00000080:2.0:1788572587.562526:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c1fafcf0 uarg=0000000000000000 00000002:00000080:2.0:1788572587.562534:0:21272:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c1fafcf0 uarg=0000000000000000 00000020:00100000:2.0:1788572587.562560:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:2.0:1788572587.562567:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940bb80 x1875453495427712/t0(0) o57->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572587.562647:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427712:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.562887:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.566424:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427712:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:2.0:1788572587.566762:0:21272:0:(file.c:6296:ll_getattr_dentry()) START file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572587.566771:0:21272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200001b71:0x4:0x0](ffff93a251194b08),name=. :VOLATILE:0000:54A7F4B8:fd=09 00000002:00010000:2.0:1788572587.566788:0:21272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x4:0x0]) in obj [0x200001b71:0x4:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572587.566801:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 30 res 8589941617/4 (0 0) 00010000:00010000:2.0:1788572587.566847:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.566857:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.566867:0:21272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 21272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.566887:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427840:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.567040:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.576190:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495427840:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.576218:0:21272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576253:0:21272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576267:0:21272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576278:0:21272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576290:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576299:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576306:0:21272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.576317:0:21272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a24940bb80 x1875453495427840/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572587.576329:0:21272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572587.576362:0:21272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576378:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/4 (0 0) 00000080:00200000:2.0:1788572587.576384:0:21272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200001b71:0x4:0x0] suppgids -1 -1: rc 0 00000080:00200000:2.0:1788572587.576440:0:21272:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x4:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788572587.576446:0:21272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) for lock 0x6d478a807bb11846 00000080:00010000:2.0:1788572587.576451:0:21272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x4:0x0](ffff93a251194b08) 00000080:00010000:2.0:1788572587.576455:0:21272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11846 from it ffffbaf6c1fafcd8 00010000:00010000:2.0:1788572587.576459:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576469:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.576505:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572587.576516:0:21272:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200001b71:0x4:0x0] 00000008:00010000:2.0:1788572587.576536:0:21272:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572587.576540:0:21272:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a2580108b0, osc lock ffff93a2489c7db0, flags 0x400001002 00010000:00010000:2.0:1788572587.576604:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.576614:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572587.576656:0:21272:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788572587.577505:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495427968:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:3.0:1788572587.583061:0:2356:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.583089:0:2356:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.583105:0:2356:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.583120:0:2356:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788572587.583157:0:2356:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788572587.583169:0:2356:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788572587.583190:0:2356:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.583200:0:2356:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.583209:0:2356:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572587.583262:0:2356:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.583554:0:21272:0:(file.c:1775:ll_merge_attr_nolock()) [0x200001b71:0x4:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788572587.583566:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba04a0 (ffff93a2516c0f38/0) at cl_lock_release():245 00000020:00010000:2.0:1788572587.583570:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba04a0 (ffff93a2516c0f38/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572587.583575:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a2580108b0 (ffff93a2516c0488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572587.583580:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba04a0 (ffff93a2516c0f38/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572587.583583:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a2580108b0 (ffff93a2516c0f38/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572587.583597:0:21272:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1788572587.583658:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=401866d8 arg=7fcba933b910 00800000:00000080:2.0:1788572587.583670:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c1fafcf0 uarg=0000000000000000 00000002:00000080:2.0:1788572587.583678:0:21272:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c1fafcf0 uarg=0000000000000000 00000020:00100000:2.0:1788572587.583706:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:2.0:1788572587.583717:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409500 x1875453495428224/t0(0) o57->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572587.583738:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495428224:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.583824:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572587.585248:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495427968:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.592551:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495428224:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:2.0:1788572587.592781:0:21272:0:(file.c:2650:do_file_write_iter()) START file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0], ppos: 0, count: 256 00000080:00200000:2.0:1788572587.592798:0:21272:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 256 bytes 00000080:00200000:2.0:1788572587.592803:0:21272:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 256 00000080:00200000:2.0:1788572587.592810:0:21272:0:(file.c:2090:ll_file_io_generic()) . :VOLATILE:0000:54A7F4B8:fd=09: write ppos: 0, bytes: 256 00000080:00200000:2.0:1788572587.592815:0:21272:0:(file.c:1848:ll_io_set_mirror()) . :VOLATILE:0000:54A7F4B8:fd=09: desiginated mirror: 0 00000020:00200000:2.0:1788572587.592823:0:21272:0:(cl_io.c:214:cl_io_rw_init()) header@000000000d0e6350[0x110, 1, [0x200001b71:0x4:0x0] hash] 00000020:00200000:2.0:1788572587.592825:0:21272:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 256) 0 0 00000080:00200000:2.0:1788572587.592830:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572587.592844:0:21272:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788572587.592850:0:21272:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 256) 256, 256 00020000:00200000:2.0:1788572587.592853:0:21272:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572587.592871:0:21272:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788572587.592893:0:21272:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572587.592898:0:21272:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a2580108b0, osc lock ffff93a2489c7db0, flags 0x0 00010000:00010000:2.0:1788572587.592913:0:21272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a258df0400 type 11 mode 2 res 9663677440/98 (0 4095) 00010000:00010000:2.0:1788572587.592961:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.592972:0:21272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.592993:0:21272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572587.593014:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495428352:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.593817:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572587.597780:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572587.597887:0:20551:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518464512 00000100:00100000:1.0:1788572587.597897:0:20551:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518464512 00000100:00100000:1.0:1788572587.598012:0:20551:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 4 00000100:00100000:1.0:1788572587.598030:0:20551:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:24342:x1875453518464512:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572587.598049:0:20551:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572587.598174:0:20551:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572587.598182:0:20551:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:24342:x1875453518464512:12345-192.168.202.102@tcp:104: Request processed in 161us (486us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572587.598189:0:20551:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 4 00010000:00010000:3.0:1788572587.604770:0:20560:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572587.604785:0:20560:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572587.604800:0:20560:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217600 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:3.0:1788572587.604809:0:20560:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572587.605069:0:20560:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572587.605083:0:20560:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572587.605137:0:20560:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 2/0,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.605145:0:20560:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572587.605182:0:20560:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 1/0,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.605191:0:20560:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000c5c9154b/0x6d478a807bb1184d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x977d9e7969c02040 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.605218:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.605220:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.605255:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572587.605517:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572587.605543:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495428480:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572587.608649:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572587.608691:0:20550:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1788572587.608709:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518464640 00000100:00100000:0.0:1788572587.608718:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518464640 00000100:00100000:0.0:1788572587.608760:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 5 00000100:00100000:0.0:1788572587.608785:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a243146850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:23644:x1875453518464640:12345-192.168.202.102@tcp:105: 00010000:00010000:0.0:1788572587.608800:0:20550:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.608813:0:20550:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.608823:0:20550:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.608835:0:20550:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.608845:0:20550:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.608858:0:20550:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000000000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.608875:0:20550:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff93a243216000) 00000100:00100000:0.0:1788572587.608926:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a243146850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:23644:x1875453518464640:12345-192.168.202.102@tcp:105: Request processed in 142us (287us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572587.608934:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 5 00000100:00100000:2.0:1788572587.610681:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495428352:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572587.610707:0:21272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.610727:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572587.610741:0:21272:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.610753:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.610762:0:21272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572587.610789:0:21272:0:(vvp_io.c:1292:vvp_io_write_start()) . :VOLATILE:0000:54A7F4B8:fd=09: write [0, 256) append 0 append lockpos 0 00000080:00200000:2.0:1788572587.610805:0:21272:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 256 bytes 00000080:00200000:2.0:1788572587.610852:0:21272:0:(rw26.c:871:ll_write_end()) pos 0, len 256, copied 256 00000080:00200000:2.0:1788572587.610861:0:21272:0:(rw26.c:929:ll_write_end()) page@00000000daf7067c[3 000000002023d1af 2 1 0000000055051654 0 0 0] 00000080:00200000:2.0:1788572587.610865:0:21272:0:(rw26.c:929:ll_write_end()) vmpage @000000009304922c 17ffffc0001001 3:0 ffff93a27425ab60 0 lru 00000080:00200000:2.0:1788572587.610872:0:21272:0:(rw26.c:929:ll_write_end()) osc-page@00000000a97cda52 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000157ef903 00000000e0bd0c5b > 3< 0 0 > 4< 0 0 8 3407872 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572587.610875:0:21272:0:(rw26.c:929:ll_write_end()) end page@00000000daf7067c 00000080:00200000:2.0:1788572587.610876:0:21272:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788572587.610881:0:21272:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 256 prio 0 00000008:00010000:2.0:1788572587.610900:0:21272:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000003e2b04d ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572587.610928:0:21272:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff93a251194cb8, count 1, dirtied 1 00000080:00200000:2.0:1788572587.610933:0:21272:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 256 bytes, tot: 256 00000080:00200000:2.0:1788572587.610935:0:21272:0:(vvp_io.c:1358:vvp_io_write_start()) . :VOLATILE:0000:54A7F4B8:fd=09: write bytes 0, result: 256 00000020:00010000:2.0:1788572587.610949:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5a88 (ffff93a2516c0f38/0) at cl_lock_release():245 00000020:00010000:2.0:1788572587.610953:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5a88 (ffff93a2516c0f38/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572587.610956:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a2580108b0 (ffff93a2516c0488/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572587.610962:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.610972:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572587.610999:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5a88 (ffff93a2516c0f38/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572587.611002:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a2580108b0 (ffff93a2516c0f38/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572587.611009:0:21272:0:(vvp_io.c:533:vvp_io_advance()) advancing 256 bytes 00000080:00200000:2.0:1788572587.611015:0:21272:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788572587.611035:0:21272:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572587.611039:0:21272:0:(file.c:2238:ll_file_io_generic()) . :VOLATILE:0000:54A7F4B8:fd=09: 2 io complete with rc: 0, result: 256, restart: 0 00000080:00200000:2.0:1788572587.611047:0:21272:0:(file.c:1775:ll_merge_attr_nolock()) [0x200001b71:0x4:0x0] updating i_size 256 i_blocks 1 00000080:00200000:2.0:1788572587.611051:0:21272:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 256 00000080:00200008:2.0:1788572587.612180:0:21272:0:(file.c:5384:ll_fsync()) START file: name=. :VOLATILE:0000:54A7F4B8:fd=09, fid=[0x200001b71:0x4:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/1 00000080:00200000:2.0:1788572587.612200:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572587.612423:0:21272:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788572587.612448:0:21272:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 0000000003e2b04d ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572587.612483:0:21272:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000003e2b04d ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.612528:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00100000:2.0:1788572587.612602:0:21272:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff93a249409500 - object 0x240000400:98 offset 0<>256 00000100:00100000:3.0:1788572587.612676:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495428608:192.168.202.102@tcp:4:lhsmtool_posix.0 00000008:00010000:2.0:1788572587.621557:0:2356:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000003e2b04d ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00010000:2.0:1788572587.621665:0:2356:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000003e2b04d ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00100000:2.0:1788572587.621692:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495428608:192.168.202.102@tcp:4:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.621720:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495428480:192.168.202.102@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1788572587.621873:0:21272:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788572587.622015:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495428736:192.168.202.102@tcp:44:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.622568:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.694485:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495428736:192.168.202.102@tcp:44:lhsmtool_posix.0 00000080:00200000:2.0:1788572587.694524:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572587.694536:0:21272:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572587.694620:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572587.694638:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495428864:192.168.202.102@tcp:16:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.733053:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495428864:192.168.202.102@tcp:16:lhsmtool_posix.0 00000080:00200000:0.0:1788572587.733136:0:21272:0:(file.c:1775:ll_merge_attr_nolock()) [0x200001b71:0x4:0x0] updating i_size 256 i_blocks 11 00000080:00200000:0.0:1788572587.733152:0:21272:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:0.0:1788572587.733171:0:21272:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x200001b71:0x4:0x0](ffff93a251194b08) cmd=c03066b0 arg=7fcba933aac0 00000080:00200008:0.0:1788572587.733194:0:21272:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08) current size 256, valid attrs 100001f0, mode 0, uid 0, gid 0, new size 0, atime 1788572585.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000 00010000:00010000:0.0:1788572587.733257:0:21272:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572587.733282:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572587.733326:0:21272:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572587.733333:0:21272:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572587.733337:0:21272:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216c00/0x6d478a807bb11846 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x977d9e7969c02039 expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572587.733354:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572587.733361:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d0e00 x1875453495429120/t0(0) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572587.733382:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495429120:192.168.202.102@tcp:36:lhsmtool_posix.0 00000100:00100000:0.0:1788572587.733466:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a247b69b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.738435:0:21272:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff93a251f1d800: committing for last_committed 17179869213 gen 1 00000100:00100000:2.0:1788572587.738455:0:21272:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a25064f800 x1875453495416576/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/4688 e 0 to 0 dl 1788572600 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572587.738501:0:21272:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a249409f80 x1875453495424768/t17179869209(17179869209) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:100 00000100:00100000:2.0:1788572587.738511:0:21272:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff93a24940a680 x1875453495426304/t17179869211(17179869211) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000100:00100000:2.0:1788572587.738523:0:21272:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff93a2460d0e00 x1875453495429120/t17179869214(17179869214) o36->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572603 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572587.738536:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0e00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495429120:192.168.202.102@tcp:36:lhsmtool_posix.0 00000080:00200000:2.0:1788572587.738615:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572587.738627:0:21272:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572587.738650:0:21272:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788572587.738662:0:21272:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572587.738666:0:21272:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff93a2580108b0, osc lock ffff93a2489c7db0, flags 0x0 00010000:00010000:2.0:1788572587.738687:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572587.738715:0:21272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572587.738730:0:21272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572587.738759:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572587.738767:0:21272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00100000:3.0:1788572587.738925:0:2355:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572587.738940:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495429248:192.168.202.102@tcp:2:lhsmtool_posix.0 00000100:00100000:3.0:1788572587.742402:0:2355:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff93a251f1d800: committing for last_committed 17179869187 gen 1 00000100:00100000:3.0:1788572587.742439:0:2355:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff93a249408380 x1875453495429248/t17179869189(17179869189) o2->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 440/432 e 0 to 0 dl 1788572603 ref 2 fl Rpc:RQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000100:00100000:3.0:1788572587.742468:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495429248:192.168.202.102@tcp:2:lhsmtool_posix.0 00000020:00010000:2.0:1788572587.742615:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff93a250ba5d58 (ffff93a2516c06c8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572587.742620:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a250ba5d58 (ffff93a2516c06c8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572587.742625:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff93a2580108b0 (ffff93a2516c0248/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572587.742637:0:21272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572587.742653:0:21272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:2.0:1788572587.742667:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a250ba5d58 (ffff93a2516c06c8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572587.742671:0:21272:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff93a2580108b0 (ffff93a2516c06c8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572587.742688:0:21272:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00800000:00000080:2.0:1788572587.742716:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d4 len=392 karg=ffff93a244b4f400 uarg=0000000000000000 00000002:00000080:2.0:1788572587.742726:0:21272:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d4 len=392 karg=ffff93a244b4f400 uarg=0000000000000000 00000020:00100000:2.0:1788572587.742757:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 55, max 7 00000100:00100000:2.0:1788572587.742765:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249408000 x1875453495429504/t0(0) o55->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 472/224 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:2.0:1788572587.742789:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495429504:192.168.202.102@tcp:55:lhsmtool_posix.0 00000100:00100000:2.0:1788572587.742895:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572587.748730:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572587.748824:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518464896 00000100:00100000:1.0:1788572587.748833:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518464896 00000100:00100000:1.0:1788572587.748886:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 6 00000100:00100000:1.0:1788572587.748896:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23656:x1875453518464896:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572587.748915:0:20550:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572587.749092:0:20550:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572587.749112:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23656:x1875453518464896:12345-192.168.202.102@tcp:104: Request processed in 216us (391us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572587.749119:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 6 00010000:00010000:2.0:1788572587.749574:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572587.749584:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749593:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216a00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572587.749599:0:20561:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749608:0:20561:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749616:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 3/0,0 mode: PR/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749667:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749675:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572587.749709:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749717:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216a00/0x6d478a807bb11838 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x4:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x977d9e7969c0201d expref: -99 pid: 21272 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.749731:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.749733:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.749735:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572587.749798:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495429632:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572587.754662:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495429632:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572587.754922:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495429504:192.168.202.102@tcp:55:lhsmtool_posix.0 00000080:00200008:0.0:1788572587.754947:0:21272:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08) current size 256, valid attrs 100001f0, mode 0, uid 0, gid 0, new size 0, atime 1788572585.000000000, mtime 1788572586.000000000, ctime 1788572586.000000000, rc 0 00000080:00200080:0.0:1788572587.754967:0:21272:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=400866d7 arg=7fcba4000b78 00000080:00200000:0.0:1788572587.755000:0:21272:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572587.755041:0:21272:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788572587.757728:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495429760:192.168.202.102@tcp:1:lhsmtool_posix.0 00000100:00100000:1.0:1788572587.761377:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572587.761545:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518465024 00000100:00100000:0.0:1788572587.761554:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518465024 00000100:00100000:0.0:1788572587.761608:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 7 00000100:00100000:0.0:1788572587.761618:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:24340:x1875453518465024:12345-192.168.202.102@tcp:104: 00010000:00010000:0.0:1788572587.761636:0:20550:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572587.761747:0:20550:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572587.761755:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:24340:x1875453518465024:12345-192.168.202.102@tcp:104: Request processed in 137us (528us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572587.761762:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 7 00010000:00010000:2.0:1788572587.763530:0:20560:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572587.763543:0:20560:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800420000020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572587.763557:0:20560:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216000 already unused, calling callback (ffffffffc11ba300) 00010000:00010000:2.0:1788572587.763564:0:20560:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800428400020000 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572587.763588:0:20560:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572587.763649:0:20560:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572587.763700:0:20560:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572587.763709:0:20560:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572587.763787:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495429888:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572587.764613:0:20560:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572587.764628:0:20560:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff93a251f1d800 lock: 00000000f33bfc5f/0x6d478a807bb11854 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804c69400020020 nid: local remote: 0x977d9e7969c02047 expref: -99 pid: 21272 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572587.764660:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.764662:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.764664:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572587.768047:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495429888:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572587.768603:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495429760:192.168.202.102@tcp:1:lhsmtool_posix.0 00800000:00000080:0.0:1788572587.768679:0:21272:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c1fafcf0 uarg=0000000000000000 00000002:00000080:0.0:1788572587.768690:0:21272:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c1fafcf0 uarg=0000000000000000 00000020:00100000:0.0:1788572587.768720:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:0.0:1788572587.768729:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d3800 x1875453495430144/t0(0) o57->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572587.768752:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495430144:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:0.0:1788572587.768828:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a247b699c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572587.772618:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:1.0:1788572587.772704:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518465152 00000100:00100000:1.0:1788572587.772714:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518465152 00000100:00100000:1.0:1788572587.772770:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 8 00000100:00100000:1.0:1788572587.772781:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a2460d3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23657:x1875453518465152:12345-192.168.202.102@tcp:104: 00010000:00010000:1.0:1788572587.772799:0:20550:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572587.772962:0:20550:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572587.772970:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a2460d3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23657:x1875453518465152:12345-192.168.202.102@tcp:104: Request processed in 191us (363us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572587.772988:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 8 00010000:00010000:2.0:1788572587.773051:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572587.773059:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572587.773087:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243216e00 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:2.0:1788572587.773093:0:20561:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572587.773100:0:20561:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572587.773109:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 3/0,0 mode: CR/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572587.773131:0:20561:0:(vvp_object.c:117:vvp_conf_set()) [0x200001b71:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1788572587.773165:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 2/0,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572587.773173:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572587.773206:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 1/0,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572587.773215:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216e00/0x6d478a807bb1183f lrc: 0/0,0 mode: --/CR res: [0x200001b71:0x4:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x977d9e7969c0202b expref: -99 pid: 21272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572587.773266:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.773268:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572587.773269:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572587.773738:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495430272:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572587.777439:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495430272:192.168.202.102@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572587.789126:0:2345:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572587.789595:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518465536 00000100:00100000:0.0:1788572587.789604:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518465536 00000100:00100000:0.0:1788572587.789656:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 9 00000100:00100000:0.0:1788572587.789666:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23657:x1875453518465536:12345-192.168.202.102@tcp:105: 00010000:00010000:0.0:1788572587.789681:0:20550:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 5/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.789728:0:20550:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 5/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.789738:0:20550:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 5/1,0 mode: --/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.789748:0:20550:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 5/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.789757:0:20550:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 6/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10080000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.789767:0:20550:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 6/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x80000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572587.789795:0:20550:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff93a243217c00) 00000100:00100000:0.0:1788572587.789852:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23657:x1875453518465536:12345-192.168.202.102@tcp:105: Request processed in 187us (736us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572587.789860:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 9 00010000:00010000:2.0:1788572587.790132:0:21271:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 4/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.790143:0:21271:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 4/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.790170:0:21271:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff93a249409c00 x1875453495425280/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 376/344 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572587.790182:0:21271:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.790196:0:21271:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 16 res 8589941617/2 (0 0) 00000080:00200000:2.0:1788572587.790212:0:21271:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200001b71:0x2:0x0] suppgids -1 -1: rc 0 00000080:00010000:2.0:1788572587.790269:0:21271:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb11823 00000080:00010000:2.0:1788572587.790287:0:21271:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11823 from it ffffbaf6c1fefc18 00010000:00010000:2.0:1788572587.790291:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.790302:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.790324:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.790335:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572587.790513:0:21271:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200001b71:0x2:0x0](0000000007d2e219) being reconfigured ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572587.790573:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495430400:192.168.202.102@tcp:49:lfs.0 00000100:00100000:2.0:1788572587.790634:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.793917:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495430400:192.168.202.102@tcp:49:lfs.0 00010000:00010000:2.0:1788572587.793994:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572587.794010:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572587.794091:0:21271:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572587.794101:0:21271:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572587.794123:0:21271:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572587.794136:0:21271:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572587.794146:0:21271:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200001b71:0x2:0x0](0000000007d2e219) being reconfigured ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572587.794192:0:21271:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572587.794196:0:21271:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572587.794435:0:21271:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:2.0:1788572587.794441:0:21271:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572587.794451:0:21271:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217c00/0x6d478a807bb11823 lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c01fc2 expref: -99 pid: 21271 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00800000:00000080:2.0:1788572587.794465:0:21271:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000002:00000080:2.0:1788572587.794486:0:21271:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d9 len=56 karg=0000000000000000 uarg=0000000000000000 00000020:00100000:2.0:1788572587.794511:0:21271:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 2 is allocated opc 58, max 7 00000100:00100000:2.0:1788572587.794519:0:21271:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409c00 x1875453495430656/t0(0) o58->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572587.794540:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495430656:192.168.202.102@tcp:58:lfs.0 00000100:00100000:2.0:1788572587.794661:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517dcc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572587.801831:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3800 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495430144:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:0.0:1788572587.801960:0:21272:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:54A7F4B8:fd=09:[0x200001b71:0x4:0x0](ffff93a251194b08), flags 500002 00000002:00100000:0.0:1788572587.802036:0:21272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940a680 x1875453495426304/t17179869211(17179869211) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000020:00100000:0.0:1788572587.802047:0:21272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572587.802052:0:21272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d1880 x1875453495430912/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:0.0:1788572587.802085:0:21272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1880 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495430912:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:0.0:1788572587.802163:0:21272:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a247b699c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.807546:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495430656:192.168.202.102@tcp:58:lfs.0 00000080:00200008:2.0:1788572587.807602:0:21271:0:(file.c:479:ll_file_release()) START file f1f.sanity-pcc:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 154002 00000002:00100000:2.0:1788572587.807667:0:21271:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249409f80 x1875453495424768/t17179869209(17179869209) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:100 00000020:00100000:2.0:1788572587.807680:0:21271:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 2 is allocated opc 35, max 7 00000100:00100000:2.0:1788572587.807686:0:21271:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409c00 x1875453495431168/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:2.0:1788572587.807715:0:21271:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495431168:192.168.202.102@tcp:35:lfs.500 00000100:00100000:2.0:1788572587.807782:0:21271:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517dcc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572587.813104:0:21272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1880 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21272:1875453495430912:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:0.0:1788572587.813136:0:21272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a24940a680 x1875453495426304/t17179869211(17179869211) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 608/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:100 00000100:00100000:2.0:1788572587.814052:0:21271:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21271:1875453495431168:192.168.202.102@tcp:35:lfs.500 00000002:00100000:2.0:1788572587.814073:0:21271:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249409f80 x1875453495424768/t17179869209(17179869209) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.500' uid:500 gid:500 projid:100 00000080:00200000:1.0:1788572587.843541:0:21274:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:1.0:1788572587.843568:0:21274:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572587.843586:0:21274:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572587.843601:0:21274:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572587.843622:0:21274:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572587.843632:0:21274:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:1.0:1788572587.843636:0:21274:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:1.0:1788572587.843645:0:21274:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c1fefb98 00010000:00010000:1.0:1788572587.843648:0:21274:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572587.843658:0:21274:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572587.843673:0:21274:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572587.843681:0:21274:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=24 00000080:00200000:1.0:1788572587.843686:0:21274:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 1 00000080:00200000:1.0:1788572587.843932:0:21274:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), flags=8 00000080:00200000:1.0:1788572587.843937:0:21274:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f1f.sanity-pcc, dir=[0x200001b71:0x1:0x0](ffff93a25110ec08), intent=getattr 00000002:00010000:1.0:1788572587.843951:0:21274:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f1f.sanity-pcc,[0x0:0x0:0x0]) in obj [0x200001b71:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572587.844009:0:21274:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21274 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572587.844087:0:21274:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21274 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572587.844098:0:21274:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21274 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572587.844129:0:21274:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21274:1875453495431296:192.168.202.102@tcp:101:lfs.0 00000100:00100000:1.0:1788572587.844768:0:21274:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a253e82780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572587.854536:0:21274:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21274:1875453495431296:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572587.854567:0:21274:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854584:0:21274:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854606:0:21274:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854618:0:21274:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854628:0:21274:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854641:0:21274:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854649:0:21274:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854657:0:21274:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572587.854669:0:21274:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff93a258da0700 x1875453495431296/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572603 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572587.854681:0:21274:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572587.854695:0:21274:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854705:0:21274:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 4 res 8589941617/2 (0 0) 00000080:00200000:2.0:1788572587.854717:0:21274:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200001b71:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572587.854760:0:21274:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x2:0x0](ffff93a251196c08): rc = 0 00000080:00200000:2.0:1788572587.854773:0:21274:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 1 -> 1 00000080:00010000:2.0:1788572587.854778:0:21274:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb1185b 00000080:00010000:2.0:1788572587.854788:0:21274:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00010000:2.0:1788572587.854792:0:21274:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1185b from it ffffbaf6c1fefb60 00010000:00010000:2.0:1788572587.854796:0:21274:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572587.854805:0:21274:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572588.311193:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572588.857763:0:2346:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.102@tcp (source: 12345-192.168.202.102@tcp) 00000100:00100000:0.0:1788572588.857878:0:20550:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875453518465664 00000100:00100000:0.0:1788572588.857888:0:20550:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875453518465664 00000100:00100000:0.0:1788572588.857957:0:20550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.102@tcp, seq: 10 00000100:00100000:0.0:1788572588.857968:0:20550:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff93a258da1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23689:x1875453518465664:12345-192.168.202.102@tcp:107:hsm_cdtr.0 00000100:00100000:0.0:1788572588.858059:0:20550:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff93a258da1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:bf89ae46-999b-46d3-9a62-08dfe43909d0+5:23689:x1875453518465664:12345-192.168.202.102@tcp:107:hsm_cdtr.0 Request processed in 92us (305us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572588.858067:0:20550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.102@tcp, seq: 10 00000080:00200080:2.0:1788572588.858574:0:21285:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=c0086696 arg=7fcba4000d20 00800000:00000080:2.0:1788572588.858597:0:21285:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=c0086696 len=4128 karg=ffff93a2491c8000 uarg=0000000000000000 00000002:00000080:2.0:1788572588.858604:0:21285:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=c0086696 len=4128 karg=ffff93a2491c8000 uarg=0000000000000000 00000002:00000080:2.0:1788572588.858609:0:21285:0:(mdc_request.c:1747:mdc_ioc_fid2path()) path get [0x200001b71:0x2:0x0] from 18446744073709551615 #0 00000100:00100000:2.0:1788572588.858643:0:21285:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495431424:192.168.202.102@tcp:53:lhsmtool_posix.0 00000100:00100000:2.0:1788572588.858732:0:21285:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572588.866623:0:21285:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495431424:192.168.202.102@tcp:53:lhsmtool_posix.0 00000002:00000080:2.0:1788572588.866652:0:21285:0:(mdc_request.c:1767:mdc_ioc_fid2path()) path got [0x200000007:0x1:0x0] from 18446744073709551615 #0: d1f.sanity-pcc/f1f.sanity-pcc 00000080:00200000:2.0:1788572588.866775:0:21285:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000002:0x2:0x0](ffff93a25110c2c8), inode mode 4040 mask 1 00000080:00200000:2.0:1788572588.866792:0:21285:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=0x200001b71:0x2:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110c2c8), file ffff93a253a3bb00, open_flags 68800, mode 0 opened 0 00000080:00200000:2.0:1788572588.866799:0:21285:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=0x200001b71:0x2:0x0, dir=[0x200000002:0x2:0x0](ffff93a25110c2c8), intent=open 00000002:00010000:2.0:1788572588.866822:0:21285:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x2:0x0]) in obj [0x200000002:0x2:0x0], intent: open flags 041504001 00010000:00010000:2.0:1788572588.866905:0:21285:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21285 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.866915:0:21285:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21285 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572588.866926:0:21285:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572588.866944:0:21285:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409c00 x1875453495431680/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572588.866954:0:21285:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21285 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572588.866984:0:21285:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495431680:192.168.202.102@tcp:101:lhsmtool_posix.0 00000100:00100000:2.0:1788572588.867154:0:21285:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572588.871142:0:21285:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495431680:192.168.202.102@tcp:101:lhsmtool_posix.0 00010000:00010000:2.0:1788572588.871512:0:21285:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 4/1,0 mode: --/CR res: [0x200000002:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871530:0:21285:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871538:0:21285:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 4/1,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871549:0:21285:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871560:0:21285:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871589:0:21285:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871597:0:21285:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 5/2,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871604:0:21285:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 4/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572588.871616:0:21285:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff93a249409c00 x1875453495431680/t17179869223(17179869223) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572604 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572588.871629:0:21285:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409c00 x1875453495431680/t17179869223(17179869223) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572604 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572588.871641:0:21285:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572588.871653:0:21285:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871666:0:21285:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572588.871678:0:21285:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 2/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572588.871686:0:21285:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871698:0:21285:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871705:0:21285:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572588.871779:0:21285:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572588.871787:0:21285:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000002:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572588.871839:0:21285:0:(namei.c:147:ll_iget()) got inode: [0x200001b71:0x2:0x0](ffff93a251196c08): rc = 0 00000080:00200000:2.0:1788572588.871855:0:21285:0:(file.c:6843:ll_layout_conf()) [0x200001b71:0x2:0x0]: layout version change: 1 -> 1 00000080:00010000:2.0:1788572588.871860:0:21285:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) for lock 0x6d478a807bb1185b 00000080:00010000:2.0:1788572588.871870:0:21285:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200001b71:0x2:0x0](ffff93a251196c08) 00000080:00200008:2.0:1788572588.871883:0:21285:0:(file.c:1090:ll_file_open()) START file 0x200001b71:0x2:0x0:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 1504000 00000002:00100000:2.0:1788572588.871904:0:21285:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a249409c00 x1875453495431680/t17179869223(17179869223) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572604 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572588.871952:0:21285:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb1185b from it ffff93a242ca0d20 00010000:00010000:2.0:1788572588.871957:0:21285:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 3/1,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572588.871966:0:21285:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572588.872000:0:21285:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x2:0x0](ffff93a251196c08), inode mode 81a4 mask 44 00000080:00200080:2.0:1788572588.873307:0:21285:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88) cmd=400866d7 arg=7fcba4000b78 00800000:00000080:2.0:1788572588.873321:0:21285:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c201fcf0 uarg=0000000000000000 00000002:00000080:2.0:1788572588.873329:0:21285:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff93a251f1d800: cmd=401866d8 len=64 karg=ffffbaf6c201fcf0 uarg=0000000000000000 00000020:00100000:2.0:1788572588.873364:0:21285:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 57, max 7 00000100:00100000:2.0:1788572588.873373:0:21285:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a249409f80 x1875453495431936/t0(0) o57->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 512/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572588.873395:0:21285:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495431936:192.168.202.102@tcp:57:lhsmtool_posix.0 00000100:00100000:2.0:1788572588.874341:0:21285:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517d3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572588.879379:0:21285:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409f80 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495431936:192.168.202.102@tcp:57:lhsmtool_posix.0 00000080:00200008:3.0:1788572588.879436:0:21285:0:(file.c:479:ll_file_release()) START file 0x200001b71:0x2:0x0:[0x200001b71:0x2:0x0](ffff93a251196c08), flags 1504000 00000002:00100000:3.0:1788572588.879499:0:21285:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a249409c00 x1875453495431680/t17179869223(17179869223) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572604 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572588.879526:0:21285:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572588.879531:0:21285:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a25064d180 x1875453495432192/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00100000:3.0:1788572588.879553:0:21285:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495432192:192.168.202.102@tcp:35:lhsmtool_posix.0 00000100:00100000:3.0:1788572588.879631:0:21285:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24867c540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572588.882195:0:21285:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job lhsmtool_posix:bf89ae46-999b-46d3-9a62-08dfe43909d0:21285:1875453495432192:192.168.202.102@tcp:35:lhsmtool_posix.0 00000002:00100000:3.0:1788572588.882266:0:21285:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff93a249409c00 x1875453495431680/t17179869223(17179869223) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572604 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lhsmtool_posix.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788572588.882807:0:20561:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572588.882818:0:20561:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572588.882837:0:20561:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff93a243217600 already unused, calling callback (ffffffffc1af2630) 00010000:00010000:0.0:1788572588.882843:0:20561:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572588.882852:0:20561:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572588.882861:0:20561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 3/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572588.883089:0:20561:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c7a00/0x6d478a807bb1185b lrc: 2/0,0 mode: PR/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x977d9e7969c02094 expref: -99 pid: 21274 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572588.883132:0:20561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 2/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572588.883139:0:20561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572588.883177:0:20561:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 1/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572588.883185:0:20561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243217600/0x6d478a807bb11862 lrc: 0/0,0 mode: --/PR res: [0x200001b71:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x977d9e7969c020a2 expref: -99 pid: 21285 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572588.883203:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572588.883222:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495432320:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572588.883494:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572588.883497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572588.883499:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572588.889728:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495432320:192.168.202.102@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572589.336069:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572590.361615:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572590.934547:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2f28dd17-6644-45dd-bf11-f8af88abec3c:2356:1875453495432448:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.934650:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495432704:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.934680:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495432960:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.934703:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495433216:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.934733:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788572590.934739:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495432576:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.934763:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495432832:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.934783:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495433088:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.940116:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495432576:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.940135:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495432832:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.940162:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495433088:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.940174:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495432704:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.940183:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495432960:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.940193:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495433216:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572590.945877:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2f28dd17-6644-45dd-bf11-f8af88abec3c:2356:1875453495432448:192.168.202.102@tcp:400:kworker.0 00000080:00200000:2.0:1788572591.079454:0:21304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88),name=/ 00000002:00010000:2.0:1788572591.079483:0:21304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572591.079500:0:21304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572591.079517:0:21304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788572591.079524:0:21304:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572591.079533:0:21304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) 00000080:00010000:2.0:1788572591.079537:0:21304:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff93a251173a88) for lock 0x6d478a807bb117ac 00000080:00010000:2.0:1788572591.079544:0:21304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb117ac from it ffffbaf6c2077bf8 00010000:00010000:2.0:1788572591.079547:0:21304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572591.079556:0:21304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a2742c6600/0x6d478a807bb117ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c01e17 expref: -99 pid: 21183 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788572591.079583:0:21304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff93a251173a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572591.079591:0:21304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d1f.sanity-pcc, flags=267 00000080:00200000:2.0:1788572591.079598:0:21304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08), inode mode 41ff mask 44 00000080:00200008:2.0:1788572591.079607:0:21304:0:(file.c:1090:ll_file_open()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00000002:00010000:2.0:1788572591.079621:0:21304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200001b71:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240304001 00010000:00010000:2.0:1788572591.079671:0:21304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21304 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572591.079678:0:21304:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21304 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572591.079689:0:21304:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572591.079701:0:21304:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a24940a300 x1875453495433472/t0(0) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00010000:00010000:2.0:1788572591.079710:0:21304:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21304 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572591.079732:0:21304:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21304:1875453495433472:192.168.202.102@tcp:101:lfs.0 00000100:00100000:2.0:1788572591.079914:0:21304:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a24517dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572591.086309:0:21304:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a300 pname:cluuid:pid:xid:nid:opc:job lfs:bf89ae46-999b-46d3-9a62-08dfe43909d0:21304:1875453495433472:192.168.202.102@tcp:101:lfs.0 00010000:00010000:2.0:1788572591.086347:0:21304:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572591.086357:0:21304:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 4/1,0 mode: --/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572591.086388:0:21304:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 4/1,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572591.086400:0:21304:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 4/1,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572591.086417:0:21304:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff93a24940a300 x1875453495433472/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572607 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:100 00000002:00100000:2.0:1788572591.086428:0:21304:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940a300 x1875453495433472/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572607 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:100 00000002:00010000:2.0:1788572591.086437:0:21304:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572591.086447:0:21304:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff93a27429f000 type 13 mode 16 res 8589941617/1 (0 0) 00000080:00200000:2.0:1788572591.086454:0:21304:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572591.086472:0:21304:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200001b71:0x1:0x0], updating i_size 16384 00000080:00010000:2.0:1788572591.086477:0:21304:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200001b71:0x1:0x0](ffff93a25110ec08) for lock 0x6d478a807bb11869 00000080:00010000:2.0:1788572591.086485:0:21304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x6d478a807bb11869 from it ffffbaf6c2077bd0 00010000:00010000:2.0:1788572591.086489:0:21304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 3/1,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572591.086498:0:21304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572591.086520:0:21304:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff93a24940a300 x1875453495433472/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572607 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:100 00000080:00200080:2.0:1788572591.086604:0:21304:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200001b71:0x1:0x0](ffff93a25110ec08) cmd=901066fc arg=7ffe1970d9c0 00000080:00200008:2.0:1788572591.086635:0:21304:0:(file.c:479:ll_file_release()) START file d1f.sanity-pcc:[0x200001b71:0x1:0x0](ffff93a25110ec08), flags 304000 00010000:00010000:2.0:1788572591.086651:0:21304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572591.392595:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572592.406923:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572593.430544:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572594.456447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572595.478468:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572596.054452:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:2.0:1788572596.054469:0:36:0:(import.c:65:import_set_state_nolock()) ffff93a24916c800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572596.054473:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495433728:192.168.202.102@tcp:400:kworker.0 00000100:02000000:2.0:1788572596.054476:0:36:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff93a251f1d800: disconnect after 22s idle 00000100:00100000:3.0:1788572596.054634:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2f28dd17-6644-45dd-bf11-f8af88abec3c:2356:1875453495433600:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572596.054682:0:2356:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572596.054690:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495433856:192.168.202.102@tcp:400:kworker.0 00000100:00080000:2.0:1788572596.061117:0:36:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:2.0:1788572596.061123:0:36:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff93a244fd4000: disconnect after 21s idle 00000100:00080000:2.0:1788572596.061132:0:36:0:(import.c:65:import_set_state_nolock()) ffff93a258ddd000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:2.0:1788572596.061134:0:36:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff93a244fd4000: disconnect after 21s idle 00000100:00100000:2.0:1788572596.061186:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495434112:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572596.061284:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495433984:192.168.202.102@tcp:9:kworker.0 00000100:00100000:1.0:1788572596.061348:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495434240:192.168.202.102@tcp:9:kworker.0 00000100:00100000:1.0:1788572596.061375:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495434368:192.168.202.102@tcp:9:kworker.0 00010000:00010000:2.0:1788572596.502446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572597.526449:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572598.553959:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572599.574425:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572600.598679:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572601.174431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572601.174435:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495434496:192.168.202.102@tcp:400:kworker.0 00000100:00080000:0.0:1788572601.174439:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572601.174450:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572601.174464:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572601.174467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572601.174470:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572601.174490:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2354:1875453495434624:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572601.174536:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:96693705-5e29-4404-974e-e904d01a815e:2354:1875453495434880:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572601.174542:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495434752:192.168.202.102@tcp:400:kworker.0 00010000:00010000:2.0:1788572601.622411:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572602.646513:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572603.670448:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572604.694478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572605.718419:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572606.294360:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a249409180 x1875453495433984/t0(0) o9->lustre-OST0001-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572606.294394:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495433984/t0(0) o9->lustre-OST0001-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572606.294425:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a24940b800 x1875453495434240/t0(0) o9->lustre-OST0000-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572606.294446:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495434240/t0(0) o9->lustre-OST0000-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572606.294448:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2355:1875453495435008:192.168.202.102@tcp:400:kworker.0 00000100:00080000:2.0:1788572606.294452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572606.294463:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572606.294469:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a249409c00 x1875453495434368/t0(0) o9->lustre-OST0001-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572606.294477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572606.294482:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572606.294484:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495434368/t0(0) o9->lustre-OST0001-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572606.294487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572606.294496:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572606.294512:0:2352:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = -110 req@ffff93a249409180 x1875453495433984/t0(0) o9->lustre-OST0001-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Interpret:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572606.294521:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495435136:192.168.202.102@tcp:400:kworker.0 00000100:00080000:1.0:1788572606.294523:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916c800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788572606.294543:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495433984:192.168.202.102@tcp:9:kworker.0 00000100:00100000:3.0:1788572606.294562:0:2355:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495435264:192.168.202.102@tcp:400:kworker.0 00000100:00080000:1.0:1788572606.294564:0:2352:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = -110 req@ffff93a24940b800 x1875453495434240/t0(0) o9->lustre-OST0000-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Interpret:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572606.294575:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572606.294578:0:2356:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495435392:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572606.294583:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495434240:192.168.202.102@tcp:9:kworker.0 00000100:00080000:1.0:1788572606.294595:0:2352:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = -110 req@ffff93a249409c00 x1875453495434368/t0(0) o9->lustre-OST0001-osc-ffff93a244fd4000@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572606 ref 1 fl Interpret:EXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572606.294609:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258ddd000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788572606.294615:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495434368:192.168.202.102@tcp:9:kworker.0 00010000:00010000:2.0:1788572606.742412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572607.766396:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572608.790401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572609.814348:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572610.838377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572611.414417:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2354:1875453495435648:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572611.414417:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495435520:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572611.414499:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495435776:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572611.414500:0:2354:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:96693705-5e29-4404-974e-e904d01a815e:2354:1875453495435904:192.168.202.102@tcp:400:kworker.0 00010000:00010000:2.0:1788572611.862372:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572612.438310:0:2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a249409880 x1875453495434112/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.446037:0:2356:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a249409880 x1875453495434112/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.446051:0:2356:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788572612.446055:0:2356:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff93a244fd4000: Connection to lustre-MDT0000 (at 192.168.202.102@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788572612.450325:0:2356:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1788572612.450355:0:2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a249408a80 x1875453495433856/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.450396:0:2356:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a249408a80 x1875453495433856/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.450402:0:2356:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788572612.450406:0:2356:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff93a251f1d800: Connection to lustre-OST0000 (at 192.168.202.102@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788572612.450411:0:2356:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1788572612.450435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:2.0:1788572612.450437:0:2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a249408e00 x1875453495433728/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572612.450443:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572612.450449:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572612.450457:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 793 00000100:00100000:2.0:1788572612.450461:0:2356:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a249408e00 x1875453495433728/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572612.450465:0:2353:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495436032:192.168.202.102@tcp:400:kworker.0 00000100:00080000:2.0:1788572612.450475:0:2356:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:00080000:0.0:1788572612.450478:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:02000400:2.0:1788572612.450484:0:2356:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff93a251f1d800: Connection to lustre-MDT0000 (at 192.168.202.102@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788572612.450489:0:2356:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000008:00100000:0.0:1788572612.450493:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00000400:2.0:1788572612.450509:0:2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff93a249408700 x1875453495433600/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572612.450512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495436288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572612.450526:0:2356:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a249408700 x1875453495433600/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.450536:0:2356:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788572612.450539:0:2356:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.102@tcp: Connection to MGS (at 192.168.202.102@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788572612.450548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572612.450552:0:2353:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:0.0:1788572612.450553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572612.450557:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572612.450561:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:0:1875453495436160:192.168.202.102@tcp:400:kworker.0 00000100:00080000:0.0:1788572612.450565:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 796 00000100:00080000:0.0:1788572612.450575:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572612.450581:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572612.450596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495436416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572612.450676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788572612.450755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495436416:192.168.202.102@tcp:38:kworker.0 00000100:00080000:2.0:1788572612.453910:0:2356:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.102@tcp for MGC192.168.202.102@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788572612.453913:0:2356:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788572612.453919:0:2356:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff93a249408700 x1875453495433600/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.453944:0:2356:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff93a2460d0700 x1875453495434496/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572617 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.453956:0:2356:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff93a24940a680 x1875453495435008/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572622 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.453967:0:2356:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff93a2460d0a80 x1875453495435520/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572627 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.453972:0:2356:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff93a2460d0000 x1875453495436032/t0(0) o400->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572628 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788572612.453984:0:2356:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572612.453986:0:2356:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572612.453987:0:2356:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788572612.453988:0:2356:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788572612.453994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572612.454001:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495436032:192.168.202.102@tcp:400:kworker.0 00000100:00080000:0.0:1788572612.454002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572612.454007:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572612.454024:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 793 00000100:00100000:1.0:1788572612.454029:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495435520:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572612.454029:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495434112:192.168.202.102@tcp:400:kworker.0 00000100:00080000:0.0:1788572612.454038:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572612.454042:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495433856:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572612.454051:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495433728:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572612.454055:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2355:1875453495435008:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572612.454058:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f28dd17-6644-45dd-bf11-f8af88abec3c:2353:1875453495434496:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.454066:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495436544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572612.454066:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2f28dd17-6644-45dd-bf11-f8af88abec3c:2356:1875453495433600:192.168.202.102@tcp:400:kworker.0 00000100:00080000:0.0:1788572612.454094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572612.454098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572612.454100:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572612.454107:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 793 00000100:00100000:1.0:1788572612.454107:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495436544:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572612.454117:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572612.454127:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572612.454142:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495436672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572612.454197:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436672:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572612.886400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572613.910384:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572614.934361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572615.958358:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572616.534423:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572616.534437:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572616.534463:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572616.982389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788572617.558295:0:2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff93a2460d1c00 x1875453495434752/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572617.558317:0:2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff93a2460d2300 x1875453495434624/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572617.558344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.558349:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572617.558350:0:2354:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a2460d2300 x1875453495434624/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572617.558362:0:2354:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff93a24916e000 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:2.0:1788572617.558367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.558370:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1788572617.558373:0:2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff93a2460d1180 x1875453495434880/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572617.558374:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.558381:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572617.558385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.558397:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572617.558401:0:2354:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a2460d1180 x1875453495434880/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572617.558410:0:2354:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff93a24a13b000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1788572617.558422:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2354:1875453495434624:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572617.558430:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:96693705-5e29-4404-974e-e904d01a815e:2354:1875453495434880:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572617.570279:0:2353:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a2460d1c00 x1875453495434752/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572617.570288:0:2353:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff93a24916a000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:1.0:1788572617.570302:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495434752:192.168.202.102@tcp:400:kworker.0 00010000:00010000:2.0:1788572618.006393:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572619.030403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572620.054385:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572621.078596:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1788572621.654377:0:8:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572621.654394:0:8:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572621.654421:0:8:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572622.102395:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572622.678353:0:2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff93a24940aa00 x1875453495435264/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678368:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff93a2460d0380 x1875453495436672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678414:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495436672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678497:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff93a2460d3480 x1875453495436544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678510:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495436544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678518:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff93a2460d2680 x1875453495436288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678530:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495436288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.678537:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff93a2460d2a00 x1875453495436416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572622.678538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678546:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572622.678549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678559:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572622.678564:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572622.678570:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495436416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572622.678585:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572622.678590:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572622.678594:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572622.678595:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572622.678617:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00080000:0.0:1788572622.678618:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678624:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788572622.678626:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436672:192.168.202.102@tcp:38:kworker.0 00000100:00080000:0.0:1788572622.678629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678637:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572622.678640:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572622.678648:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572622.678673:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 834 00000100:00080000:3.0:1788572622.678675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572622.678681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495436544:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572622.678687:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572622.678688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572622.678699:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:3.0:1788572622.678701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572622.678708:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436288:192.168.202.102@tcp:8:kworker.0 00000008:00100000:0.0:1788572622.678715:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788572622.678719:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572622.678728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572622.678731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495436416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788572622.678746:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495436800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572622.678761:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572622.678768:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572622.678770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436800:192.168.202.102@tcp:38:kworker.0 00000100:00080000:0.0:1788572622.678775:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 834 00000100:00080000:0.0:1788572622.678780:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572622.678785:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572622.678791:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572622.678804:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495436928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572622.678822:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678825:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572622.678827:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572622.678831:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 834 00000100:00080000:0.0:1788572622.678837:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572622.678841:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572622.678847:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572622.678861:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495437056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572622.678864:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436928:192.168.202.102@tcp:8:kworker.0 00000100:00080000:0.0:1788572622.678886:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.678890:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572622.678892:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572622.678896:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 834 00000100:00080000:0.0:1788572622.678902:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572622.678906:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572622.678921:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495437184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572622.678933:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495437056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572622.678974:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495437184:192.168.202.102@tcp:250:kworker.0 00000100:00000400:3.0:1788572622.679037:0:2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff93a24940ad80 x1875453495435136/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.679051:0:2356:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a24940ad80 x1875453495435136/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572622.679057:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff93a24916e000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:3.0:1788572622.679062:0:2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff93a249408000 x1875453495435392/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572622.679072:0:2356:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a249408000 x1875453495435392/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572622.679077:0:2356:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff93a24a13b000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:3.0:1788572622.679083:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2356:1875453495435136:192.168.202.102@tcp:400:kworker.0 00000100:00100000:3.0:1788572622.679087:0:2356:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:96693705-5e29-4404-974e-e904d01a815e:2356:1875453495435392:192.168.202.102@tcp:400:kworker.0 00000100:00100000:2.0:1788572622.691533:0:2355:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a24940aa00 x1875453495435264/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572622.691548:0:2355:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff93a24916a000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:2.0:1788572622.691593:0:2355:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2355:1875453495435264:192.168.202.102@tcp:400:kworker.0 00010000:00010000:2.0:1788572623.126374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572624.150400:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572625.174401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572626.198380:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572627.222434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788572627.798343:0:2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572611/real 1788572611] req@ffff93a2460d1f80 x1875453495435776/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572627.798355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572627.798363:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572627.798367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572627.798371:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572627.798375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572627.798383:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572627.798388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572627.798392:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1788572627.798417:0:2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572611/real 1788572611] req@ffff93a2460d1500 x1875453495435648/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572627.798445:0:2354:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a2460d1500 x1875453495435648/t0(0) o400->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572627.798455:0:2354:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff93a24916e000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1788572627.798460:0:2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572611/real 1788572611] req@ffff93a2460d3b80 x1875453495435904/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572627.798471:0:2354:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a2460d3b80 x1875453495435904/t0(0) o400->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572627.798478:0:2354:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff93a24a13b000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1788572627.798488:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:bf89ae46-999b-46d3-9a62-08dfe43909d0:2354:1875453495435648:192.168.202.102@tcp:400:kworker.0 00000100:00100000:0.0:1788572627.798494:0:2354:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:96693705-5e29-4404-974e-e904d01a815e:2354:1875453495435904:192.168.202.102@tcp:400:kworker.0 00000100:00100000:1.0:1788572627.807377:0:2353:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff93a2460d1f80 x1875453495435776/t0(0) o400->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572627.807390:0:2353:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff93a24916a000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1788572627.807418:0:2353:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:bf89ae46-999b-46d3-9a62-08dfe43909d0:2353:1875453495435776:192.168.202.102@tcp:400:kworker.0 00010000:00010000:2.0:1788572628.246387:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572629.270421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572630.294456:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572631.318440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572632.342460:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572632.918402:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.918412:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.918416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.918421:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.918425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.918452:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.918455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.918458:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1788572633.366426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572634.390462:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572635.414403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1788572635.990433:0:2344:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.202.102@tcp (192.168.202.102:988) timed out; the network or node may be down. 00010000:00010000:2.0:1788572636.438456:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572637.014391:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff93a2460d2680 x1875453495437184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014467:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495437184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014483:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff93a2460d2a00 x1875453495437056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014500:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495437056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014513:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff93a2460d2300 x1875453495436928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014535:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495436928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014549:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff93a2460d1180 x1875453495436800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572637.014563:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495436800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572637.014629:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572637.014655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572637.014661:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495437184:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572637.014672:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572637.014682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572637.014691:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572637.014697:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572637.014702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00080000:2.0:1788572637.014709:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 844 00000100:00100000:1.0:1788572637.014713:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495437056:192.168.202.102@tcp:38:kworker.0 00000100:00080000:2.0:1788572637.014721:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572637.014724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572637.014731:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:1.0:1788572637.014742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572637.014749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436928:192.168.202.102@tcp:8:kworker.0 00000100:00080000:1.0:1788572637.014761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788572637.014764:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495437312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572637.014785:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572637.014788:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572637.014790:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572637.014794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00080000:2.0:1788572637.014798:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 844 00000100:00080000:2.0:1788572637.014803:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 10s 00000100:00100000:1.0:1788572637.014809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495436800:192.168.202.102@tcp:38:kworker.0 00000100:00080000:2.0:1788572637.014814:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572637.014824:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572637.014834:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495437440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572637.014843:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572637.014857:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572637.014859:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572637.014867:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 844 00000100:00080000:2.0:1788572637.014875:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788572637.014878:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572637.014882:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572637.014895:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495437312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572637.014895:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495437568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572637.014922:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572637.014924:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572637.014925:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572637.014928:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 844 00000100:00080000:2.0:1788572637.014930:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788572637.014933:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572637.014936:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572637.014944:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495437696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572637.015040:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495437440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788572637.015090:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495437568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788572637.015130:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495437696:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572637.462391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572638.487139:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572639.510798:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572640.535183:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572641.558690:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572642.134326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.134337:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.134340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.134344:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.134347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.134351:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.134355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.134359:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000400:00000100:2.0:1788572642.390581:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.102:988 00000400:00000100:2.0:1788572642.390624:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572642.390649:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572642.390691:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572642.390720:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572642.390731:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788572642.390752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572642] req@ffff93a249408e00 x1875453495437568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.390780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495437568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.390805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.390844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.390850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495437568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788572642.390859:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572642] req@ffff93a249408a80 x1875453495437696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.390875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495437696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.390884:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.390891:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.390895:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495437696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788572642.390902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572642] req@ffff93a24940aa00 x1875453495437312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.390916:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495437312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.390926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.390931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.390934:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495437312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788572642.390942:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572642] req@ffff93a249408700 x1875453495437440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.390955:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495437440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.390972:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.390977:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.390979:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495437440:192.168.202.102@tcp:38:kworker.0 00000100:00080000:3.0:1788572642.391403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.391409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572642.391413:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572642.391419:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 858 00000100:00080000:3.0:1788572642.391429:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572642.391438:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572642.391487:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495437824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572642.391508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.391512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572642.391514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572642.391518:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 858 00000100:00080000:3.0:1788572642.391530:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572642.391536:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572642.391546:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572642.391560:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495437952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572642.391571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.391574:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572642.391577:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572642.391581:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 858 00000100:00080000:3.0:1788572642.391585:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572642.391588:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572642.391593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572642.391604:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495438080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572642.391614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.391617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572642.391620:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572642.391623:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 858 00000100:00080000:3.0:1788572642.391627:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572642.391631:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572642.391636:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572642.391648:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495438208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572642.391796:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495437824:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788572642.391859:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572642.391893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572642/real 1788572642] req@ffff93a25064d180 x1875453495437824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.391909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495437824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.391918:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.391922:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.391925:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495437824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788572642.391933:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495437952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788572642.391956:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572642.391969:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572642/real 1788572642] req@ffff93a25064fb80 x1875453495437952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.391982:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495437952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.391991:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.391995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.391998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495437952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788572642.392013:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438080:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788572642.392037:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572642.392074:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572642/real 1788572642] req@ffff93a25064e300 x1875453495438080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.392087:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495438080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.392096:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.392100:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.392103:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438080:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788572642.392110:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495438208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788572642.392133:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572642.392143:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572642/real 1788572642] req@ffff93a25064c380 x1875453495438208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572642.392156:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495438208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572642.392162:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572642.392164:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788572642.392166:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495438208:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572642.582575:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572643.606473:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572644.630823:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572645.654659:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572646.678825:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572647.702436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572647.766315:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572647.766323:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572647.766326:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572647.766347:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 863 00000100:00080000:1.0:1788572647.766355:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572647.766361:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572647.766390:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495438336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572647.766413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572647.766416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572647.766417:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572647.766420:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 863 00000100:00080000:1.0:1788572647.766423:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572647.766426:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572647.766435:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572647.766445:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495438336:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788572647.766446:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495438464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572647.766460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572647.766463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572647.766464:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572647.766467:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 863 00000100:00080000:1.0:1788572647.766470:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572647.766472:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572647.766475:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572647.766484:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495438592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572647.766501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572647.766503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572647.766504:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572647.766508:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 863 00000100:00080000:1.0:1788572647.766512:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572647.766516:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572647.766521:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572647.766535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495438720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572647.766560:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572647.766590:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572647.766614:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495438720:192.168.202.102@tcp:38:kworker.0 00000400:00000100:0.0:1788572648.534534:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.102:988 00000400:00000100:0.0:1788572648.534604:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572648.534622:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572648.534669:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572648.534681:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572648.534695:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572648.534722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572648] req@ffff93a258da1180 x1875453495438464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572648.534756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495438464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572648.534772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572648.534779:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572648.534784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572648.534794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572648] req@ffff93a258da0000 x1875453495438592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572648.534807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495438592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572648.534815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572648.534819:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572648.534847:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572648.534855:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572648] req@ffff93a258da2300 x1875453495438720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572648.534885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495438720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572648.534894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572648.534898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572648.534901:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495438720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572648.534909:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572648] req@ffff93a258da3100 x1875453495438336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572648.534924:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495438336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572648.534933:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572648.534937:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572648.534941:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495438336:192.168.202.102@tcp:250:kworker.0 00010000:00010000:3.0:1788572648.726891:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572649.750904:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572650.774740:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572651.798781:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572652.823540:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572652.886301:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572652.886310:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572652.886314:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572652.886323:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 869 00000100:00080000:1.0:1788572652.886331:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572652.886338:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572652.886369:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495438848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572652.886390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572652.886393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572652.886394:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572652.886397:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 869 00000100:00080000:1.0:1788572652.886400:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572652.886402:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572652.886423:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572652.886433:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495438976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572652.886448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572652.886452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788572652.886452:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495438848:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572652.886456:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572652.886464:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 869 00000100:00080000:1.0:1788572652.886467:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572652.886469:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572652.886473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572652.886482:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495439104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572652.886490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572652.886493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572652.886494:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572652.886497:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 869 00000100:00080000:1.0:1788572652.886500:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572652.886502:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572652.886506:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572652.886513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495439232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572652.886563:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572652.886604:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495439104:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572652.886625:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495439232:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572653.846547:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788572654.678572:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.102:988 00000400:00000100:0.0:1788572654.678608:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572654.678622:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572654.678660:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572654.678669:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572654.678692:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572654.678731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572652/real 1788572654] req@ffff93a258da3480 x1875453495438976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572654.678769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495438976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572654.678783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572654.678789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572654.678794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495438976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572654.678804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572652/real 1788572654] req@ffff93a258da2a00 x1875453495439104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572654.678817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495439104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572654.678825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572654.678829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572654.678832:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495439104:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572654.678841:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572652/real 1788572654] req@ffff93a258da0380 x1875453495439232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572654.678854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495439232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572654.678884:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572654.678889:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572654.678892:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495439232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572654.678900:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572652/real 1788572654] req@ffff93a258da0700 x1875453495438848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572654.678912:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495438848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572654.678920:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572654.678923:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572654.678927:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495438848:192.168.202.102@tcp:250:kworker.0 00010000:00010000:3.0:1788572654.871086:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572655.894725:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572656.918837:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572657.942687:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572658.006361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572658.006369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572658.006373:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572658.006380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 874 00000100:00080000:2.0:1788572658.006389:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572658.006394:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572658.006436:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495439360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572658.006461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572658.006467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572658.006469:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572658.006472:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 874 00000100:00080000:2.0:1788572658.006474:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572658.006477:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572658.006485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572658.006494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495439488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572658.006503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572658.006504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495439360:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572658.006520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572658.006522:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572658.006526:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 874 00000100:00080000:2.0:1788572658.006529:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572658.006532:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572658.006536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572658.006544:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495439616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572658.006551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572658.006553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572658.006555:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788572658.006558:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572658.006562:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 874 00000100:00080000:2.0:1788572658.006566:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572658.006568:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572658.006572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572658.006592:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495439744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572658.006621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572658] req@ffff93a249409880 x1875453495439360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572658.006642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495439360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572658.006663:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572658.006670:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572658.006675:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495439360:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572658.006694:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495439488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572658.006717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572658.006726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572658] req@ffff93a249409500 x1875453495439488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572658.006734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495439488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572658.006738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572658.006741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572658.006742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495439488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572658.006746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495439616:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572658.006756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572658.006761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572658] req@ffff93a24940b100 x1875453495439616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572658.006768:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495439616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572658.006772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572658.006773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572658.006775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495439616:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572658.006778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495439744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572658.006786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572658.006792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572658] req@ffff93a24940bb80 x1875453495439744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572658.006807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495439744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572658.006814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572658.006817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572658.006820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495439744:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572658.967342:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572659.990967:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572661.014419:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572662.039003:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572663.063552:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572663.126328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572663.126337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572663.126342:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572663.126351:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 879 00000100:00080000:0.0:1788572663.126360:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572663.126367:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572663.126398:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495439872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572663.126452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572663.126454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572663.126455:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572663.126459:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 879 00000100:00080000:0.0:1788572663.126462:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572663.126466:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572663.126475:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572663.126488:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495440000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572663.126493:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495439872:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572663.126503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572663.126508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572663.126510:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572663.126514:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 879 00000100:00080000:0.0:1788572663.126517:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572663.126519:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572663.126524:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572663.126532:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495440128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572663.126539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572663.126542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572663.126543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572663.126545:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 879 00000100:00080000:0.0:1788572663.126552:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572663.126555:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572663.126560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572663.126570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495440256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572663.126592:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572663.126616:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572663.126631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495440256:192.168.202.102@tcp:38:kworker.0 00000400:00000100:1.0:1788572663.894509:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.102:988 00000400:00000100:1.0:1788572663.894554:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572663.894580:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788572663.894618:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788572663.894627:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788572663.894643:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572663.894656:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572663] req@ffff93a2460d1500 x1875453495440000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572663.894677:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495440000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572663.894686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572663.894691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572663.894695:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572663.894708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572663] req@ffff93a2460d3480 x1875453495440128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572663.894741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495440128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572663.894749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572663.894753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572663.894757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572663.894763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572663] req@ffff93a2460d0380 x1875453495440256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572663.894775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495440256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572663.894782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572663.894786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572663.894789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495440256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572663.894795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572663] req@ffff93a2460d3b80 x1875453495439872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572663.894806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495439872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572663.894813:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572663.894815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572663.894818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495439872:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788572664.086538:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572665.110751:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572666.134492:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572667.159673:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572668.183111:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572668.246381:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572668.246390:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572668.246410:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572668.246418:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 884 00000100:00080000:2.0:1788572668.246427:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572668.246433:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572668.246463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495440384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572668.246486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572668.246489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572668.246490:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572668.246493:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 884 00000100:00080000:2.0:1788572668.246496:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572668.246498:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572668.246505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572668.246524:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495440512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572668.246531:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495440384:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572668.246535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572668.246539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572668.246540:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572668.246543:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 884 00000100:00080000:2.0:1788572668.246545:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572668.246547:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572668.246554:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572668.246564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495440640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572668.246581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572668.246583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572668.246585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572668.246587:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 884 00000100:00080000:2.0:1788572668.246591:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572668.246592:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572668.246597:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572668.246601:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572668.246613:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495440768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572668.246646:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff93a24940b480 x1875453495440384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.246669:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495440384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.246681:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.246685:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572668.246688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495440384:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572668.246707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572668.246726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572668.246746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff93a24940bb80 x1875453495440512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.246755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495440512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.246760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.246763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572668.246765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572668.246769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440640:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572668.246789:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572668.246801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff93a24940b100 x1875453495440640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.246825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495440640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.246834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.246847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572668.246850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495440640:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572668.246854:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495440768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572668.246871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572668.246881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff93a249409500 x1875453495440768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.246894:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495440768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.246901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.246905:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572668.246908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495440768:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788572669.208694:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572670.230575:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572671.255154:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572672.278512:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572673.302639:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572673.366354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572673.366363:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572673.366366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572673.366374:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 889 00000100:00080000:0.0:1788572673.366383:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572673.366388:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572673.366420:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495440896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572673.366441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572673.366444:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572673.366445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572673.366448:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 889 00000100:00080000:0.0:1788572673.366450:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572673.366452:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572673.366459:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572673.366467:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495441024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572673.366473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572673.366488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572673.366490:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572673.366493:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 889 00000100:00080000:0.0:1788572673.366496:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572673.366500:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572673.366505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572673.366513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495441152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572673.366521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572673.366525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1788572673.366525:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495440896:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572673.366528:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572673.366536:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 889 00000100:00080000:0.0:1788572673.366538:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572673.366540:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572673.366543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572673.366552:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495441280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572673.366620:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572673.366648:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441152:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572673.366664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495441280:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572674.327127:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572675.350568:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788572676.182547:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.102:988 00000400:00000100:2.0:1788572676.182593:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572676.182611:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572676.182663:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572676.182675:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572676.182687:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572676.182712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff93a2460d0a80 x1875453495441152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.182747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495441152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.182760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.182765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572676.182769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441152:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572676.182779:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff93a2460d0000 x1875453495441280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.182793:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495441280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.182802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.182806:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572676.182810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495441280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572676.182822:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff93a2460d1c00 x1875453495440896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.182840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495440896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.182848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.182851:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572676.182854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495440896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572676.182876:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff93a2460d0700 x1875453495441024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.182889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495441024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.182901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.182904:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572676.182908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441024:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572676.374481:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572677.399305:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572678.422897:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572678.486331:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572678.486342:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572678.486347:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572678.486360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 894 00000100:00080000:0.0:1788572678.486392:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572678.486400:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572678.486437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495441408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572678.486467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572678.486470:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572678.486472:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572678.486475:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 894 00000100:00080000:0.0:1788572678.486478:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572678.486481:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572678.486489:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572678.486500:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495441536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572678.486513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572678.486516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572678.486518:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572678.486522:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495441408:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572678.486523:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 894 00000100:00080000:0.0:1788572678.486529:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572678.486536:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572678.486539:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572678.486548:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495441664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572678.486555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572678.486563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572678.486569:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572678.486572:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 894 00000100:00080000:0.0:1788572678.486576:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572678.486580:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572678.486584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572678.486592:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788572678.486597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495441792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572678.486649:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff93a2460d0700 x1875453495441408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.486666:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495441408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.486675:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.486679:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572678.486682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495441408:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572678.486701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572678.486724:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572678.486739:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff93a2460d1c00 x1875453495441536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.486753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495441536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.486770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.486773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572678.486777:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572678.486795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441664:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572678.486817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572678.486826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff93a2460d0000 x1875453495441664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.486834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495441664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.486839:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.486842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572678.486844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495441664:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572678.486848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495441792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572678.486860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572678.486866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff93a2460d0a80 x1875453495441792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.486873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495441792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.486878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.486880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572678.486888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495441792:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572679.446487:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572680.471330:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572681.494439:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572682.518434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572683.542686:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572683.606316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572683.606325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572683.606335:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572683.606344:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 900 00000100:00080000:1.0:1788572683.606353:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572683.606359:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572683.606396:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495441920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572683.606436:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572683.606441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572683.606444:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572683.606449:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 900 00000100:00080000:1.0:1788572683.606454:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572683.606459:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572683.606468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572683.606484:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495442048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572683.606484:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495441920:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572683.606501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572683.606504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572683.606505:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572683.606521:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 900 00000100:00080000:1.0:1788572683.606525:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572683.606528:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572683.606532:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572683.606550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495442176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572683.606559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788572683.606560:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788572683.606563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572683.606565:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572683.606569:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 900 00000100:00080000:1.0:1788572683.606572:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572683.606575:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572683.606579:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572683.606587:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495442304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572683.606615:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff93a258da1c00 x1875453495441920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.606650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495441920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.606661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.606664:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572683.606667:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495441920:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572683.606693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572683.606715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572683.606724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff93a258da1880 x1875453495442048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.606733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495442048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.606739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.606741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572683.606743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572683.606747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442176:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572683.606759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572683.606765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff93a258da0a80 x1875453495442176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.606772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495442176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.606786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.606788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572683.606791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442176:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572683.606808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495442304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572683.606826:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572683.606834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff93a258da0e00 x1875453495442304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.606842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495442304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.606847:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.606849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572683.606851:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495442304:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572684.566652:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572685.591490:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572686.614789:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572687.639357:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572688.662465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572688.726387:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572688.726394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572688.726397:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572688.726406:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 905 00000100:00080000:0.0:1788572688.726414:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572688.726426:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572688.726456:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495442432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572688.726493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572688.726496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572688.726498:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572688.726503:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 905 00000100:00080000:0.0:1788572688.726508:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572688.726512:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572688.726522:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572688.726544:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495442432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572688.726547:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495442560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572688.726557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572688.726560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572688.726562:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572688.726566:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 905 00000100:00080000:0.0:1788572688.726569:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572688.726573:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572688.726578:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572688.726589:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495442688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572688.726598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572688.726602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572688.726603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788572688.726604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572688.726607:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 905 00000100:00080000:0.0:1788572688.726616:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572688.726620:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572688.726625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572688.726635:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495442816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572688.726642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff93a2460d3b80 x1875453495442432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.726659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495442432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.726686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.726690:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572688.726695:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495442432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572688.726724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572688.726751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572688.726761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff93a2460d0380 x1875453495442560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.726769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495442560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.726776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.726779:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572688.726783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572688.726789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442688:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572688.726811:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572688.726823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff93a2460d3480 x1875453495442688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.726835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495442688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.726844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.726848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572688.726851:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495442688:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572688.726857:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495442816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572688.726872:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572688.726879:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff93a2460d1500 x1875453495442816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.726893:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495442816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.726898:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.726900:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572688.726902:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495442816:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572689.687526:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572690.710459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572691.735327:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572692.758607:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572693.783048:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572693.846366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.846375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572693.846379:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.846387:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 910 00000100:00080000:1.0:1788572693.846396:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.846402:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572693.846434:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495442944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.846456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.846471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572693.846473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.846476:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 910 00000100:00080000:1.0:1788572693.846479:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.846481:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572693.846490:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572693.846501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495443072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.846510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.846513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572693.846515:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.846517:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 910 00000100:00080000:1.0:1788572693.846519:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.846522:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572693.846525:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572693.846531:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495443200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.846550:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.846553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572693.846554:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.846558:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 910 00000100:00080000:1.0:1788572693.846563:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.846569:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572693.846569:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495442944:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788572693.846583:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572693.846595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495443328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572693.846729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572693.846784:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443200:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572693.846841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495443328:192.168.202.102@tcp:38:kworker.0 00000400:00000100:0.0:1788572694.614544:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.102:988 00000400:00000100:0.0:1788572694.614598:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572694.614614:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572694.614658:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572694.614668:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572694.614676:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:2.0:1788572694.614757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572694] req@ffff93a258da3800 x1875453495443200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572694.614782:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495443200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572694.614795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572694.614800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572694.614805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443200:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572694.614814:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572694] req@ffff93a258da0e00 x1875453495443328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572694.614828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495443328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572694.614849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572694.614854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572694.614858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495443328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572694.614866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572694] req@ffff93a258da1f80 x1875453495442944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572694.614881:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495442944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572694.614890:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572694.614894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572694.614898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495442944:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572694.614905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572694] req@ffff93a258da2680 x1875453495443072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572694.614917:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495443072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572694.614925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572694.614929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572694.614932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443072:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572694.807041:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572695.830879:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572696.854659:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572697.879022:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572698.902671:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572698.966390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572698.966403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572698.966407:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572698.966452:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 915 00000100:00080000:2.0:1788572698.966481:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572698.966490:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572698.966526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495443456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572698.966563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572698.966568:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572698.966570:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572698.966576:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 915 00000100:00080000:2.0:1788572698.966580:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572698.966584:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572698.966595:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572698.966610:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495443584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572698.966626:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495443456:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572698.966630:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572698.966637:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572698.966640:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572698.966647:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 915 00000100:00080000:2.0:1788572698.966651:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572698.966658:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572698.966665:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572698.966682:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495443712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572698.966695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572698.966707:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572698.966710:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572698.966718:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 915 00000800:00000100:3.0:1788572698.966721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572698.966727:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572698.966735:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572698.966743:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572698.966760:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495443840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572698.966793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff93a249409880 x1875453495443456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.966817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495443456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.966827:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.966831:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572698.966834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495443456:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572698.966859:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572698.966879:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572698.966889:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff93a249408700 x1875453495443584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.966897:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495443584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.966903:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.966905:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572698.966908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572698.966912:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443712:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572698.966924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572698.966945:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff93a24940aa00 x1875453495443712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.966953:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495443712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.966959:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.966961:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572698.966964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495443712:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572698.966967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495443840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572698.966980:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572698.966987:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff93a249408a80 x1875453495443840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.966994:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495443840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.966999:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.967001:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572698.967003:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495443840:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572699.927217:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572700.950793:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572701.974731:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572702.998749:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572704.022525:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572704.086310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572704.086321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572704.086325:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572704.086338:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 920 00000100:00080000:2.0:1788572704.086350:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572704.086358:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572704.086402:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495443968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572704.086429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572704.086432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572704.086434:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572704.086437:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 920 00000100:00080000:2.0:1788572704.086457:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572704.086461:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572704.086472:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572704.086484:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495444096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572704.086492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572704.086495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572704.086496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572704.086500:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 920 00000100:00080000:2.0:1788572704.086503:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572704.086506:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572704.086510:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572704.086517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495444224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572704.086527:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495443968:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572704.086529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572704.086538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572704.086548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572704.086553:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 920 00000100:00080000:2.0:1788572704.086555:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572704.086557:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572704.086560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572704.086567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495444352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572704.086622:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572704.086681:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff93a249408e00 x1875453495443968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.086697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495443968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.086721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.086725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572704.086728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495443968:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572704.086735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572704.086755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572704.086765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff93a249408000 x1875453495444096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.086774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495444096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.086779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.086781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572704.086783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572704.086804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444224:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572704.086817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572704.086824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff93a24940ad80 x1875453495444224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.086832:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495444224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.086837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.086839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572704.086842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444224:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572704.086851:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495444352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572704.086866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572704.086873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff93a24940a680 x1875453495444352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.086881:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495444352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.086894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.086896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572704.086898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495444352:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572705.047012:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572706.070728:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572707.095353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572708.119155:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572709.142793:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572709.206309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572709.206319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572709.206323:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572709.206344:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 925 00000100:00080000:2.0:1788572709.206355:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572709.206363:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572709.206405:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495444480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572709.206430:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572709.206434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572709.206436:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572709.206440:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 925 00000100:00080000:2.0:1788572709.206445:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572709.206450:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572709.206461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572709.206474:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495444480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572709.206478:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495444608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572709.206491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572709.206494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572709.206496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572709.206516:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 925 00000100:00080000:2.0:1788572709.206521:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572709.206526:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572709.206533:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572709.206539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572709.206550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495444736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572709.206562:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572709.206566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572709.206568:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572709.206572:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 925 00000100:00080000:2.0:1788572709.206576:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572709.206580:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572709.206585:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572709.206595:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff93a249409c00 x1875453495444480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572709.206600:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495444864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572709.206637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495444480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.206646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.206651:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572709.206654:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495444480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572709.206673:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572709.206691:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572709.206707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff93a24940b800 x1875453495444608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.206716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495444608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.206722:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.206724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572709.206726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572709.206730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444736:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572709.206743:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572709.206749:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff93a249409180 x1875453495444736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.206756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495444736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.206761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.206763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572709.206765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495444736:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572709.206768:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495444864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572709.206778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572709.206784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff93a24940a680 x1875453495444864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.206792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495444864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.206801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.206804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572709.206806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495444864:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788572710.167259:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572711.191087:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572712.214850:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572713.238712:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572714.262842:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572714.326375:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.326387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572714.326392:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.326404:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 930 00000100:00080000:0.0:1788572714.326440:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.326449:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572714.326488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495444992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572714.326519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.326522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572714.326524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.326527:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 930 00000100:00080000:0.0:1788572714.326530:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.326533:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572714.326543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572714.326551:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495445120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572714.326559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.326561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572714.326563:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.326566:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 930 00000100:00080000:0.0:1788572714.326567:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.326569:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572714.326573:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572714.326580:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495445248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572714.326587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.326595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572714.326598:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.326601:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 930 00000100:00080000:0.0:1788572714.326604:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.326607:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572714.326611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572714.326620:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495445376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572714.326628:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495444992:192.168.202.102@tcp:250:kworker.0 00000800:00000100:3.0:1788572714.326698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572714.326746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff93a2460d1180 x1875453495444992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.326762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495444992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.326771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.326774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572714.326778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495444992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572714.326783:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572714.326805:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572714.326821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff93a2460d2300 x1875453495445120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.326832:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495445120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.326862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.326866:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572714.326870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572714.326876:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445248:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572714.326900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572714.326913:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff93a2460d2a00 x1875453495445248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.326923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495445248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.326930:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.326933:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572714.326936:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445248:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572714.326961:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495445376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572714.326993:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572714.327006:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff93a2460d2680 x1875453495445376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.327018:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495445376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.327024:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.327026:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572714.327061:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495445376:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572715.287531:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572716.310443:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572717.334453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572718.358371:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572719.382496:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572719.446364:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572719.446379:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572719.446385:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572719.446410:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 935 00000100:00080000:0.0:1788572719.446425:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572719.446435:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572719.446488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495445504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572719.446516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572719.446519:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572719.446521:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572719.446525:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 935 00000100:00080000:0.0:1788572719.446528:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572719.446532:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572719.446560:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572719.446574:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495445504:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572719.446576:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495445632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572719.446602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572719.446605:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572719.446607:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572719.446613:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 935 00000100:00080000:0.0:1788572719.446616:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572719.446620:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572719.446624:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572719.446635:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495445760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572719.446644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572719.446647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572719.446648:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788572719.446655:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572719.446657:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 935 00000100:00080000:0.0:1788572719.446664:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572719.446669:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572719.446675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572719.446686:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495445888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572719.446706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff93a2460d1f80 x1875453495445504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.446725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495445504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.446735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.446739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572719.446742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495445504:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572719.446762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572719.446786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572719.446807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff93a2460d2680 x1875453495445632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.446821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495445632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.446830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.446834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572719.446837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572719.446845:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445760:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572719.446883:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572719.446895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff93a2460d2a00 x1875453495445760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.446920:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495445760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.446926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.446929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572719.446931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495445760:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572719.446936:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495445888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572719.446952:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572719.446959:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff93a2460d2300 x1875453495445888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.446966:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495445888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.446971:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.446973:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572719.446975:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495445888:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788572720.406620:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572721.431642:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572722.455614:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572723.478593:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572724.502723:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572724.566351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.566362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572724.566367:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.566378:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 941 00000100:00080000:2.0:1788572724.566389:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.566398:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572724.566441:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495446016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572724.566468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.566489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572724.566491:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.566496:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 941 00000100:00080000:2.0:1788572724.566499:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.566503:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572724.566514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572724.566520:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495446016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572724.566528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495446144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572724.566542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.566546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572724.566548:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.566554:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 941 00000100:00080000:2.0:1788572724.566558:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.566562:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572724.566568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572724.566583:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495446272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572724.566592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572724.566597:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572724.566599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572724.566601:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.566605:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 941 00000100:00080000:2.0:1788572724.566611:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.566613:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572724.566626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572724.566635:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495446400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572724.566661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff93a24940ad80 x1875453495446016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.566678:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495446016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.566687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.566690:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572724.566694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495446016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572724.566718:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572724.566735:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572724.566744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff93a249408000 x1875453495446144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.566752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495446144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.566758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.566760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572724.566762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572724.566765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446272:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572724.566808:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572724.566817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff93a249408e00 x1875453495446272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.566825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495446272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.566854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.566857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572724.566859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446272:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572724.566864:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495446400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572724.566880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572724.566888:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff93a249408a80 x1875453495446400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.566895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495446400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.566901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.566903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572724.566905:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495446400:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572725.528358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572726.551320:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572727.574523:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572728.598881:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572729.622687:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572729.686361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572729.686371:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572729.686375:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572729.686382:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 946 00000100:00080000:1.0:1788572729.686392:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572729.686398:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572729.686435:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495446528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572729.686456:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572729.686459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572729.686460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572729.686463:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 946 00000100:00080000:1.0:1788572729.686466:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572729.686468:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572729.686476:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572729.686484:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495446656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572729.686491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572729.686493:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572729.686495:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572729.686497:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 946 00000100:00100000:3.0:1788572729.686511:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495446528:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572729.686515:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572729.686522:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572729.686526:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572729.686535:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495446784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572729.686542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572729.686545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572729.686547:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572729.686550:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 946 00000100:00080000:1.0:1788572729.686552:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572729.686554:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572729.686558:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572729.686564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495446912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572729.686619:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572729.686681:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572729.686712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495446912:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788572730.646594:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788572731.478501:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788572731.478544:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572731.478562:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572731.478619:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572731.478636:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572731.478645:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572731.478715:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572731] req@ffff93a258da1c00 x1875453495446784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572731.478753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495446784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572731.478766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572731.478771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572731.478776:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572731.478786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572731] req@ffff93a258da0700 x1875453495446912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572731.478800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495446912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572731.478809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572731.478813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572731.478817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495446912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572731.478823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572731] req@ffff93a258da0a80 x1875453495446528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572731.478837:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495446528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572731.478845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572731.478849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572731.478853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495446528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572731.478860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572731] req@ffff93a258da1880 x1875453495446656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572731.478882:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495446656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572731.478891:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572731.478895:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572731.478898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495446656:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572731.671263:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572732.694681:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572733.719509:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572734.742697:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572734.806363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572734.806372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572734.806376:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572734.806384:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 951 00000100:00080000:2.0:1788572734.806393:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572734.806399:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572734.806438:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495447040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572734.806467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572734.806471:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572734.806473:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572734.806478:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 951 00000100:00080000:2.0:1788572734.806482:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572734.806487:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572734.806497:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572734.806509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495447168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572734.806514:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495447040:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572734.806532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572734.806537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572734.806539:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572734.806544:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 951 00000100:00080000:2.0:1788572734.806575:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572734.806581:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572734.806587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572734.806602:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572734.806604:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495447296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572734.806616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572734.806619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572734.806621:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572734.806624:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 951 00000100:00080000:2.0:1788572734.806628:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572734.806630:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572734.806634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572734.806644:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495447424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572734.806654:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff93a24940aa00 x1875453495447040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.806676:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495447040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.806688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.806692:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572734.806697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495447040:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572734.806719:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572734.806739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572734.806749:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff93a249408700 x1875453495447168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.806757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495447168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.806769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.806771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572734.806774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572734.806778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447296:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572734.806794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572734.806801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff93a249409880 x1875453495447296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.806808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495447296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.806813:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.806815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572734.806817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447296:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572734.806821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495447424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572734.806832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572734.806842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff93a249409500 x1875453495447424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.806854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495447424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.806863:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.806867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572734.806876:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495447424:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572735.766907:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572736.790458:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572737.814437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572738.838451:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572739.862376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572739.926579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.926598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572739.926604:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.926620:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 956 00000100:00080000:3.0:1788572739.926633:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.926644:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572739.926697:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495447552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572739.926735:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.926739:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572739.926742:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.926746:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 956 00000100:00080000:3.0:1788572739.926751:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.926755:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572739.926768:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572739.926784:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495447680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572739.926797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.926803:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572739.926808:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572739.926812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495447552:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572739.926818:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 956 00000100:00080000:3.0:1788572739.926828:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.926853:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572739.926874:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572739.926891:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495447808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572739.926903:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.926911:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1788572739.926912:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788572739.926915:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.926925:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 956 00000100:00080000:3.0:1788572739.926934:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.926940:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572739.926946:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572739.926964:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495447936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572739.926988:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff93a25064f480 x1875453495447552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572739.927011:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495447552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572739.927025:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572739.927030:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572739.927050:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495447552:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572739.927078:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572739.927108:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572739.927123:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff93a25064ce00 x1875453495447680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572739.927136:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495447680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572739.927144:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572739.927148:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572739.927151:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572739.927158:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447808:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572739.927180:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572739.927192:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff93a25064f100 x1875453495447808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572739.927205:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495447808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572739.927213:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572739.927217:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572739.927220:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495447808:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572739.927273:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495447936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572739.927304:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572739.927329:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff93a25064d500 x1875453495447936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572739.927344:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495447936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572739.927353:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572739.927357:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572739.927361:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495447936:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572740.886453:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572741.910466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572742.934456:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572743.958451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572744.982487:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572745.046426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572745.046443:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572745.046449:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572745.046469:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 961 00000100:00080000:1.0:1788572745.046484:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572745.046492:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572745.046538:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495448064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572745.046575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572745.046601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572745.046603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572745.046609:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 961 00000100:00080000:1.0:1788572745.046613:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572745.046624:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572745.046633:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495448064:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788572745.046636:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572745.046652:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495448192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572745.046661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572745.046664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572745.046665:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572745.046669:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 961 00000100:00080000:1.0:1788572745.046672:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572745.046675:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572745.046679:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572745.046688:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495448320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572745.046696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572745.046699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572745.046701:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572745.046703:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 961 00000100:00080000:1.0:1788572745.046706:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572745.046708:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:2.0:1788572745.046712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788572745.046720:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572745.046730:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495448448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572745.046778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff93a258da0380 x1875453495448064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572745.046805:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495448064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572745.046819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572745.046824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572745.046829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495448064:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572745.046858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572745.046878:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572745.046888:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff93a258da2a00 x1875453495448192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572745.046909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495448192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572745.046916:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572745.046918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572745.046920:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572745.046924:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448320:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572745.046939:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572745.046953:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff93a258da3480 x1875453495448320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572745.046961:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495448320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572745.046966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572745.046969:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572745.046971:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448320:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572745.046974:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495448448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572745.046986:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572745.046992:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff93a258da3100 x1875453495448448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572745.046999:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495448448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572745.047014:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572745.047016:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572745.047019:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495448448:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572746.006413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572747.030434:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572748.054397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572749.078431:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572750.102421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572750.166333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.166349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572750.166356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.166365:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 966 00000100:00080000:2.0:1788572750.166373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.166379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572750.166411:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495448576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.166434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.166437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572750.166439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.166441:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 966 00000100:00080000:2.0:1788572750.166444:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.166446:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572750.166453:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572750.166461:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495448704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.166468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.166485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572750.166487:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.166491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 966 00000100:00080000:2.0:1788572750.166493:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.166496:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572750.166500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572750.166509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495448832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.166528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572750.166528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495448576:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572750.166539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572750.166542:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.166548:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 966 00000100:00080000:2.0:1788572750.166552:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.166557:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572750.166564:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572750.166576:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495448960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572750.166632:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572750.166691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff93a24940b100 x1875453495448576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.166715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495448576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.166725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.166728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572750.166732:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495448576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572750.166752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572750.166782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572750.166794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff93a24940bb80 x1875453495448704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.166803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495448704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.166809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.166811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572750.166814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572750.166848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448832:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572750.166872:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572750.166886:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff93a24940b480 x1875453495448832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.166899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495448832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.166908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.166911:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572750.166914:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495448832:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572750.166920:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495448960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572750.166937:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572750.166945:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff93a249409500 x1875453495448960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.166960:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495448960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.166965:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.166968:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572750.166970:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495448960:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572751.126386:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572752.150427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572753.174387:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572754.198414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572755.222380:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572755.286309:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572755.286317:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572755.286321:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572755.286337:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 971 00000100:00080000:2.0:1788572755.286364:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572755.286369:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572755.286399:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495449088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572755.286421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572755.286423:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572755.286425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572755.286428:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 971 00000100:00080000:2.0:1788572755.286430:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572755.286433:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572755.286440:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572755.286449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495449216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572755.286457:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495449088:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572755.286460:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572755.286464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572755.286466:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572755.286469:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 971 00000100:00080000:2.0:1788572755.286471:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572755.286473:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572755.286476:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572755.286485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495449344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572755.286492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572755.286499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572755.286500:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572755.286503:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 971 00000100:00080000:2.0:1788572755.286508:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572755.286510:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572755.286514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572755.286519:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572755.286529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495449472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572755.286567:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff93a249409880 x1875453495449088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.286582:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495449088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.286590:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.286593:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572755.286596:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495449088:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572755.286613:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572755.286628:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572755.286637:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff93a249408700 x1875453495449216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.286645:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495449216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.286661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.286663:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572755.286665:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572755.286670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449344:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572755.286684:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572755.286691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff93a24940aa00 x1875453495449344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.286699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495449344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.286704:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.286706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572755.286708:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449344:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572755.286711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495449472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572755.286722:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572755.286737:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff93a249408a80 x1875453495449472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.286745:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495449472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.286750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.286751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572755.286760:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495449472:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572756.246415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572757.270390:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572758.294355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572759.318403:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572760.342415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572760.406328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.406338:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572760.406352:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.406363:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 976 00000100:00080000:2.0:1788572760.406374:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.406381:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572760.406422:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495449600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572760.406452:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.406456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572760.406458:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.406463:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 976 00000100:00080000:2.0:1788572760.406468:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.406474:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572760.406485:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572760.406489:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495449600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572760.406519:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495449728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572760.406532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.406536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572760.406538:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.406543:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 976 00000100:00080000:2.0:1788572760.406548:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572760.406555:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572760.406556:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572760.406566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572760.406581:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495449856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572760.406593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.406597:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572760.406599:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572760.406604:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff93a249408e00 x1875453495449600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572760.406607:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 976 00000100:00080000:2.0:1788572760.406614:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.406623:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572760.406629:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572760.406633:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495449600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572760.406645:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495449984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572760.406647:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.406657:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572760.406666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495449600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572760.406694:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572760.406720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572760.406731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff93a249408000 x1875453495449728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.406741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495449728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.406747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.406749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572760.406751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572760.406757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449856:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572760.406777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572760.406798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff93a24940ad80 x1875453495449856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.406809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495449856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.406815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.406817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572760.406819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495449856:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572760.406823:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495449984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572760.406838:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572760.406846:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff93a24940a680 x1875453495449984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.406867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495449984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.406875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.406879:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572760.406882:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495449984:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572761.366408:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572762.390444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572763.414384:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572764.438409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572765.462406:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572765.526353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572765.526362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572765.526366:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572765.526381:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 981 00000100:00080000:3.0:1788572765.526390:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572765.526396:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572765.526442:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495450112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572765.526466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572765.526469:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572765.526471:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572765.526474:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 981 00000100:00080000:3.0:1788572765.526477:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572765.526479:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572765.526488:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572765.526497:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495450240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572765.526505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572765.526511:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495450112:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572765.526521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572765.526523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572765.526527:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 981 00000100:00080000:3.0:1788572765.526529:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572765.526533:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572765.526536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572765.526544:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495450368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572765.526552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572765.526555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572765.526556:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572765.526563:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 981 00000100:00080000:3.0:1788572765.526567:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572765.526570:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572765.526574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572765.526579:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572765.526585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495450496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572765.526627:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff93a25064df80 x1875453495450112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572765.526656:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495450112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572765.526668:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572765.526673:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572765.526677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495450112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572765.526701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572765.526727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572765.526741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff93a25064ca80 x1875453495450240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572765.526755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495450240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572765.526764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572765.526767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572765.526771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572765.526778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450368:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572765.526809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572765.526826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff93a25064c700 x1875453495450368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572765.526840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495450368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572765.526861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572765.526865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572765.526869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450368:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572765.526876:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495450496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572765.526899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572765.526911:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff93a25064d500 x1875453495450496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572765.526923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495450496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572765.526931:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572765.526934:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572765.526937:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495450496:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572766.486359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572767.510457:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572768.534408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572769.558388:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572770.582462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572770.646395:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572770.646410:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572770.646415:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572770.646431:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 987 00000100:00080000:3.0:1788572770.646445:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572770.646456:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572770.646509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495450624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572770.646545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572770.646550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572770.646552:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572770.646557:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 987 00000100:00080000:3.0:1788572770.646561:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572770.646566:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572770.646579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572770.646596:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495450752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572770.646608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572770.646637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788572770.646638:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495450624:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572770.646641:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572770.646656:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 987 00000100:00080000:3.0:1788572770.646661:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572770.646668:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572770.646676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572770.646695:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495450880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572770.646711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788572770.646714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788572770.646722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572770.646725:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572770.646732:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 987 00000100:00080000:3.0:1788572770.646739:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572770.646744:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572770.646750:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572770.646759:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff93a25064f100 x1875453495450624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572770.646770:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495451008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572770.646783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495450624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572770.646806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572770.646810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572770.646818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495450624:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572770.646843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572770.646871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572770.646886:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff93a25064ce00 x1875453495450752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572770.646899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495450752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572770.646908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572770.646912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572770.646915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572770.646943:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450880:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572770.646962:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572770.646970:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff93a25064f480 x1875453495450880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572770.646978:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495450880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572770.646984:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572770.646986:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572770.646989:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495450880:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572770.646993:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495451008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572770.647004:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572770.647010:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff93a25064c380 x1875453495451008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572770.647025:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495451008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572770.647030:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572770.647032:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572770.647034:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495451008:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572771.606433:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572772.630478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572773.654505:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572774.678369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572775.702463:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572775.766342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572775.766352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572775.766355:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572775.766368:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 992 00000100:00080000:3.0:1788572775.766376:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572775.766382:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572775.766419:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495451136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572775.766459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572775.766462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572775.766463:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572775.766467:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 992 00000100:00080000:3.0:1788572775.766470:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572775.766472:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572775.766484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572775.766495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495451264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572775.766502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572775.766504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572775.766510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572775.766515:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 992 00000100:00080000:3.0:1788572775.766518:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572775.766520:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495451136:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572775.766522:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572775.766529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572775.766541:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495451392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572775.766548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572775.766550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572775.766552:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572775.766555:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 992 00000100:00080000:3.0:1788572775.766562:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572775.766570:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572775.766574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572775.766583:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495451520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572775.766604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572775.766657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff93a25064e300 x1875453495451136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572775.766677:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495451136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572775.766688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572775.766693:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572775.766697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495451136:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572775.766706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572775.766731:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572775.766745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff93a25064fb80 x1875453495451264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572775.766759:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495451264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572775.766782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572775.766786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572775.766790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572775.766817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451392:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572775.766841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572775.766864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff93a25064d180 x1875453495451392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572775.766878:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495451392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572775.766887:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572775.766890:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572775.766894:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451392:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572775.766901:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495451520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572775.766921:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572775.766932:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff93a25064c000 x1875453495451520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572775.766944:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495451520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572775.766953:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572775.766956:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572775.766967:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495451520:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572776.726374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572777.750409:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572778.774375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572779.798438:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572780.822480:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572780.886395:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572780.886411:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572780.886417:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572780.886431:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 997 00000100:00080000:3.0:1788572780.886446:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572780.886454:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572780.886501:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495451648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572780.886530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572780.886533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572780.886534:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572780.886537:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 997 00000100:00080000:3.0:1788572780.886540:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572780.886543:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572780.886553:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572780.886562:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495451776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572780.886569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572780.886596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572780.886597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572780.886603:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 997 00000100:00080000:3.0:1788572780.886608:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572780.886613:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572780.886621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572780.886627:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495451648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572780.886639:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495451904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572780.886654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572780.886658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572780.886661:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572780.886666:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 997 00000100:00080000:3.0:1788572780.886670:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572780.886674:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572780.886681:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572780.886694:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495452032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572780.886714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572780.886764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff93a25064d880 x1875453495451648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572780.886784:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495451648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572780.886797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572780.886802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572780.886806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495451648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572780.886816:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572780.886841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572780.886854:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff93a25064e680 x1875453495451776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572780.886866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495451776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572780.886874:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572780.886878:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572780.886881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572780.886908:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451904:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572780.886930:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572780.886958:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff93a25064ed80 x1875453495451904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572780.886972:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495451904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572780.886981:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572780.886984:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572780.886988:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495451904:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572780.886995:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495452032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572780.887038:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572780.887051:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff93a25064dc00 x1875453495452032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572780.887065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495452032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572780.887073:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572780.887078:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572780.887081:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495452032:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572781.846452:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572782.870392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572783.894367:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572784.918402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572785.942394:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572786.006366:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572786.006373:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572786.006377:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572786.006385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1002 00000100:00080000:0.0:1788572786.006393:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572786.006400:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572786.006431:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495452160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572786.006452:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572786.006468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572786.006470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572786.006474:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1002 00000100:00080000:0.0:1788572786.006477:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572786.006481:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572786.006490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572786.006495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495452160:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572786.006503:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495452288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572786.006511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572786.006514:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572786.006515:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572786.006518:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1002 00000100:00080000:0.0:1788572786.006521:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572786.006523:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572786.006527:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572786.006535:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495452416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572786.006542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572786.006545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572786.006546:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788572786.006553:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572786.006555:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1002 00000100:00080000:0.0:1788572786.006559:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572786.006563:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572786.006574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572786.006584:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495452544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572786.006593:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff93a2460d1180 x1875453495452160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572786.006617:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495452160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572786.006632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572786.006636:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572786.006639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495452160:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572786.006658:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572786.006677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572786.006686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff93a2460d1500 x1875453495452288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572786.006694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495452288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572786.006706:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572786.006709:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572786.006711:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572786.006716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452416:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572786.006730:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572786.006740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff93a2460d3480 x1875453495452416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572786.006751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495452416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572786.006760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572786.006764:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572786.006767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452416:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572786.006791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495452544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572786.006819:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572786.006832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff93a2460d0380 x1875453495452544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572786.006844:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495452544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572786.006853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572786.006857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572786.006861:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495452544:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572786.966366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572787.990407:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572789.014515:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572790.038512:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572791.062477:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572791.126346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572791.126358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572791.126364:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572791.126385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1007 00000100:00080000:0.0:1788572791.126399:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572791.126409:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572791.126463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495452672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572791.126494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572791.126497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572791.126498:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572791.126502:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1007 00000100:00080000:0.0:1788572791.126506:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572791.126510:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572791.126523:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572791.126542:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495452800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572791.126544:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495452672:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572791.126587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572791.126590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572791.126592:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572791.126598:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1007 00000100:00080000:0.0:1788572791.126602:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572791.126604:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572791.126609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572791.126621:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495452928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572791.126629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572791.126632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572791.126634:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572791.126636:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1007 00000100:00080000:0.0:1788572791.126639:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788572791.126643:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572791.126647:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572791.126651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572791.126660:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495453056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572791.126703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff93a2460d3b80 x1875453495452672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572791.126721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495452672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572791.126730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572791.126734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572791.126738:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495452672:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572791.126765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572791.126802:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572791.126816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff93a2460d0a80 x1875453495452800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572791.126850:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495452800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572791.126856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572791.126858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572791.126861:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572791.126871:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452928:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572791.126889:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572791.126897:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff93a2460d0000 x1875453495452928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572791.126905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495452928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572791.126910:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572791.126913:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572791.126915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495452928:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572791.126919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495453056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572791.126930:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572791.126937:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff93a2460d1c00 x1875453495453056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572791.126948:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495453056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572791.126982:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572791.126986:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572791.126990:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495453056:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572792.086473:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572793.110490:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572794.134451:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572795.158478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572796.183367:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572796.246342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.246350:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572796.246353:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.246361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1012 00000100:00080000:2.0:1788572796.246370:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.246375:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572796.246405:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495453184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.246427:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.246430:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572796.246431:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.246434:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1012 00000100:00080000:2.0:1788572796.246436:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.246438:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572796.246446:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572796.246454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495453312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.246462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.246467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788572796.246467:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495453184:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572796.246481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.246486:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1012 00000100:00080000:2.0:1788572796.246489:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.246492:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572796.246497:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572796.246507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495453440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.246515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.246517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572796.246518:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.246521:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1012 00000100:00080000:2.0:1788572796.246524:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.246526:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572796.246531:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572796.246543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495453568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572796.246565:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572796.246592:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572796.246607:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495453568:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572797.208318:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572798.230450:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572799.254460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788572799.318503:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788572799.318546:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572799.318563:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788572799.318599:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788572799.318608:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788572799.318616:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572799.318652:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff93a24940b800 x1875453495453312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.318702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495453312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.318716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.318722:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572799.318726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572799.318735:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff93a249409c00 x1875453495453440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.318749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495453440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.318756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.318760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572799.318763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572799.318771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff93a24940a680 x1875453495453568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.318782:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495453568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.318789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.318793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572799.318796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495453568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572799.318833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff93a249409180 x1875453495453184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.318847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495453184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.318854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.318857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572799.318860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495453184:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788572800.278994:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572801.302406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572801.366331:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.366339:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572801.366343:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.366360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1017 00000100:00080000:2.0:1788572801.366382:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.366390:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572801.366427:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495453696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.366454:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.366458:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572801.366460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.366464:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1017 00000100:00080000:2.0:1788572801.366468:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.366472:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572801.366481:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572801.366494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495453824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.366504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.366508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572801.366511:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.366515:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1017 00000100:00080000:2.0:1788572801.366525:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.366530:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572801.366540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572801.366541:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495453696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572801.366561:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495453952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.366576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.366588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572801.366590:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.366596:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1017 00000100:00080000:2.0:1788572801.366601:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.366605:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572801.366612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572801.366623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495454080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572801.366654:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572801.366748:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff93a24940ad80 x1875453495453696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.366774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495453696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.366789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.366795:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572801.366800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495453696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572801.366809:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572801.366837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572801.366850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff93a249408000 x1875453495453824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.366863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495453824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.366885:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.366889:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572801.366892:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572801.366934:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453952:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572801.366959:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572801.366970:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff93a249408e00 x1875453495453952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.366982:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495453952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.366991:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.366995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572801.366998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495453952:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572801.367005:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495454080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572801.367028:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572801.367041:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff93a249408a80 x1875453495454080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.367053:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495454080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.367061:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.367064:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572801.367074:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495454080:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572802.326390:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572803.350413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572804.374365:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572805.398388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572806.422438:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572806.486358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572806.486371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572806.486383:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572806.486397:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1022 00000100:00080000:3.0:1788572806.486407:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572806.486416:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572806.486458:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495454208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572806.486487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572806.486490:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572806.486492:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572806.486495:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1022 00000100:00080000:3.0:1788572806.486498:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572806.486501:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572806.486512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572806.486524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495454336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572806.486534:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495454208:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572806.486538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572806.486545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572806.486547:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572806.486556:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1022 00000100:00080000:3.0:1788572806.486560:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572806.486577:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572806.486583:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572806.486593:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495454464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572806.486605:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572806.486608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572806.486610:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788572806.486618:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788572806.486626:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1022 00000100:00080000:3.0:1788572806.486635:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572806.486641:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572806.486647:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572806.486663:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495454592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572806.486674:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff93a25064ea00 x1875453495454208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.486707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495454208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.486717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.486721:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572806.486724:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495454208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572806.486746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572806.486766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572806.486788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff93a25064dc00 x1875453495454336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.486806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495454336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.486815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.486817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572806.486820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572806.486827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454464:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572806.486851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572806.486863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff93a25064ed80 x1875453495454464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.486877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495454464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.486885:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.486888:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572806.486890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454464:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572806.486895:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495454592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572806.486912:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572806.486920:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff93a25064e680 x1875453495454592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.486928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495454592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.486938:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.486941:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572806.486943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495454592:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572807.446381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572808.470406:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572809.494420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572810.518375:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572811.542401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572811.606353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.606365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572811.606369:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.606380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1028 00000100:00080000:3.0:1788572811.606389:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.606396:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572811.606430:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495454720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.606459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.606463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572811.606479:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.606486:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1028 00000100:00080000:3.0:1788572811.606491:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.606499:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572811.606500:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495454720:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788572811.606510:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572811.606524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495454848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.606533:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.606537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572811.606539:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.606545:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1028 00000100:00080000:3.0:1788572811.606559:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.606565:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:2.0:1788572811.606571:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788572811.606572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572811.606590:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495454976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.606602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.606606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572811.606608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.606613:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1028 00000100:00100000:2.0:1788572811.606617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff93a25064d880 x1875453495454720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572811.606618:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.606626:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572811.606635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572811.606642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495454720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572811.606652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788572811.606654:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495455104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572811.606658:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572811.606667:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495454720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572811.606689:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572811.606707:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572811.606715:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff93a25064c000 x1875453495454848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572811.606724:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495454848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572811.606736:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572811.606738:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572811.606741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572811.606745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454976:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572811.606760:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572811.606766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff93a25064d180 x1875453495454976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572811.606773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495454976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572811.606779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572811.606781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572811.606783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495454976:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572811.606786:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495455104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572811.606797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572811.606802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff93a25064fb80 x1875453495455104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572811.606809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495455104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572811.606814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572811.606816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572811.606822:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495455104:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572812.566465:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572813.590449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572814.614387:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572815.638396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572816.662487:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572816.726357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.726388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572816.726393:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.726405:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1033 00000100:00080000:1.0:1788572816.726418:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.726425:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572816.726471:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495455232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572816.726504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.726509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572816.726511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.726516:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1033 00000100:00080000:1.0:1788572816.726520:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.726525:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572816.726538:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572816.726559:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495455232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788572816.726573:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495455360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572816.726582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.726585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572816.726587:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.726591:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1033 00000100:00080000:1.0:1788572816.726594:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.726597:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572816.726601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572816.726610:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495455488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572816.726617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.726620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572816.726622:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.726625:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1033 00000100:00080000:1.0:1788572816.726629:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.726632:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572816.726634:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572816.726641:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572816.726645:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495455616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572816.726691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff93a258da2300 x1875453495455232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572816.726708:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495455232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572816.726717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572816.726720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572816.726724:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495455232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572816.726748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495455360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572816.726774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572816.726788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff93a258da0000 x1875453495455360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572816.726802:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495455360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572816.726811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572816.726815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572816.726831:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495455360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572816.726838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495455488:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572816.726864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572816.726878:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff93a258da1180 x1875453495455488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572816.726891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495455488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572816.726900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572816.726903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572816.726907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495455488:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572816.726914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495455616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572816.726936:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572816.726947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff93a258da3100 x1875453495455616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572816.726955:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495455616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572816.726961:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572816.726963:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572816.726965:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495455616:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572817.686504:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572818.710429:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572819.734412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572820.758519:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572821.782417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572821.846354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.846362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572821.846365:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.846374:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1038 00000100:00080000:0.0:1788572821.846384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.846389:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572821.846421:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495455744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.846446:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.846449:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572821.846450:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.846453:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1038 00000100:00080000:0.0:1788572821.846455:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.846457:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572821.846477:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572821.846488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495455872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.846495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.846498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572821.846499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.846502:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1038 00000100:00080000:0.0:1788572821.846505:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.846508:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572821.846512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572821.846530:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495455744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572821.846536:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495456000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.846551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.846555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572821.846558:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.846562:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1038 00000100:00080000:0.0:1788572821.846568:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.846572:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572821.846579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572821.846591:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788572821.846594:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495456128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572821.846628:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff93a2460d0700 x1875453495455744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572821.846642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495455744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572821.846650:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572821.846653:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572821.846656:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495455744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572821.846660:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495455872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572821.846674:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572821.846682:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff93a2460d1c00 x1875453495455872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572821.846690:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495455872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572821.846695:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572821.846696:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572821.846698:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495455872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572821.846716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456000:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572821.846738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572821.846745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff93a2460d0000 x1875453495456000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572821.846753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495456000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572821.846758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572821.846759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572821.846761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456000:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572821.846765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495456128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572821.846774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572821.846780:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff93a2460d0a80 x1875453495456128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572821.846787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495456128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572821.846792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572821.846793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572821.846795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495456128:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572822.806389:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572823.830404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572824.854408:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572825.878335:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572826.902463:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572826.966388:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572826.966412:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572826.966416:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572826.966425:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1043 00000100:00080000:2.0:1788572826.966433:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572826.966439:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572826.966468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495456256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572826.966494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572826.966497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572826.966498:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572826.966501:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1043 00000100:00080000:2.0:1788572826.966504:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572826.966506:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572826.966514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572826.966522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495456384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572826.966529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572826.966532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572826.966533:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572826.966536:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1043 00000100:00080000:2.0:1788572826.966538:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572826.966540:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572826.966543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572826.966549:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495456512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572826.966575:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495456256:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572826.966595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572826.966598:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572826.966599:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572826.966603:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1043 00000100:00080000:2.0:1788572826.966606:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572826.966609:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572826.966614:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572826.966625:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495456640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572826.966665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572826.966720:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff93a24940aa00 x1875453495456256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.966739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495456256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.966749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.966754:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572826.966757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495456256:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572826.966763:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572826.966780:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572826.966790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff93a249408700 x1875453495456384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.966799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495456384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.966804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.966806:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572826.966808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572826.966832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456512:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572826.966862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572826.966870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff93a249409880 x1875453495456512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.966879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495456512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.966885:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.966887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572826.966889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456512:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572826.966893:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495456640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572826.966904:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572826.966914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff93a249409500 x1875453495456640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.966926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495456640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.966934:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.966938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572826.966941:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495456640:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572827.926422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572828.950373:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572829.974400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572830.998406:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572832.022421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572832.086325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.086335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572832.086339:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.086349:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1048 00000100:00080000:2.0:1788572832.086359:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.086366:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572832.086414:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495456768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572832.086434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.086437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572832.086452:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.086456:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1048 00000100:00080000:2.0:1788572832.086459:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.086463:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572832.086471:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572832.086482:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495456896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572832.086491:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495456768:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572832.086492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.086498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572832.086502:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.086505:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1048 00000100:00080000:2.0:1788572832.086507:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.086509:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572832.086512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572832.086521:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495457024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572832.086528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.086531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572832.086532:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.086535:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1048 00000100:00080000:2.0:1788572832.086544:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.086547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572832.086549:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572832.086557:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572832.086567:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495457152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572832.086614:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff93a24940b480 x1875453495456768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.086641:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495456768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.086649:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.086652:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572832.086655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495456768:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572832.086701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572832.086721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572832.086731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff93a24940bb80 x1875453495456896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.086739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495456896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.086745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.086747:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572832.086749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495456896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572832.086753:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457024:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572832.086764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572832.086770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff93a24940b100 x1875453495457024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.086777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495457024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.086791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.086793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572832.086796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457024:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572832.086799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495457152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572832.086812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572832.086819:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff93a249409500 x1875453495457152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.086826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495457152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.086831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.086832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572832.086834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495457152:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572833.046382:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572834.070401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572835.094465:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572836.118415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572837.142362:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572837.206342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.206351:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572837.206354:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.206362:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1053 00000100:00080000:0.0:1788572837.206371:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.206377:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572837.206409:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495457280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572837.206430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.206433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572837.206434:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.206437:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1053 00000100:00080000:0.0:1788572837.206440:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.206460:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572837.206471:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495457280:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788572837.206472:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572837.206492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495457408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572837.206504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.206506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572837.206508:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.206513:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1053 00000100:00080000:0.0:1788572837.206519:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.206524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572837.206530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572837.206544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788572837.206545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495457536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572837.206560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.206563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572837.206564:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.206568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1053 00000100:00080000:0.0:1788572837.206571:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.206574:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572837.206579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572837.206586:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff93a2460d3b80 x1875453495457280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572837.206591:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495457664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572837.206618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495457280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.206627:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.206631:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572837.206634:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495457280:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572837.206661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572837.206679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572837.206689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff93a2460d0380 x1875453495457408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.206697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495457408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.206704:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.206718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572837.206722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572837.206729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457536:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572837.206753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572837.206797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff93a2460d3480 x1875453495457536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.206811:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495457536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.206820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.206824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572837.206827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457536:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572837.206834:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495457664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572837.206856:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572837.206867:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff93a2460d1500 x1875453495457664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.206882:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495457664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.206888:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.206890:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572837.206892:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495457664:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572838.166542:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572839.190445:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572840.214517:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572841.238387:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572842.262385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572842.326296:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572842.326320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572842.326323:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572842.326332:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1058 00000100:00080000:0.0:1788572842.326340:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572842.326346:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572842.326374:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495457792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572842.326396:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572842.326399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572842.326400:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572842.326403:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1058 00000100:00080000:0.0:1788572842.326405:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572842.326407:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572842.326415:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572842.326424:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495457920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572842.326431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572842.326434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572842.326439:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572842.326443:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1058 00000100:00080000:0.0:1788572842.326445:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572842.326448:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572842.326451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572842.326457:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495457792:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572842.326461:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495458048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572842.326476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572842.326479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572842.326480:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572842.326483:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1058 00000100:00080000:0.0:1788572842.326487:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572842.326489:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572842.326493:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572842.326502:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495458176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572842.326516:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572842.326566:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff93a2460d1180 x1875453495457792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572842.326591:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495457792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572842.326603:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572842.326607:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572842.326612:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495457792:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572842.326619:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572842.326638:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572842.326647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff93a2460d2300 x1875453495457920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572842.326656:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495457920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572842.326672:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572842.326674:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572842.326677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495457920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572842.326695:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458048:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572842.326715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572842.326728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff93a2460d2a00 x1875453495458048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572842.326741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495458048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572842.326750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572842.326753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572842.326757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458048:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572842.326764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495458176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572842.326785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572842.326798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff93a2460d2680 x1875453495458176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572842.326811:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495458176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572842.326818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572842.326820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572842.326828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495458176:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572843.286394:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572844.310417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572845.334429:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572846.358385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572847.382352:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572847.446332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572847.446343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572847.446347:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572847.446378:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1063 00000100:00080000:1.0:1788572847.446387:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572847.446393:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572847.446425:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495458304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572847.446446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572847.446448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572847.446450:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572847.446453:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1063 00000100:00080000:1.0:1788572847.446455:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572847.446458:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572847.446466:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572847.446474:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495458432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572847.446483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572847.446485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572847.446486:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572847.446491:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1063 00000100:00080000:1.0:1788572847.446493:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572847.446494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495458304:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572847.446496:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572847.446503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572847.446510:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495458560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572847.446528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572847.446530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572847.446532:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572847.446535:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1063 00000100:00080000:1.0:1788572847.446538:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572847.446541:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572847.446545:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572847.446553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495458688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572847.446575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572847.446620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff93a258da3480 x1875453495458304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572847.446638:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495458304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572847.446646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572847.446649:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572847.446652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495458304:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572847.446670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572847.446687:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572847.446696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff93a258da2a00 x1875453495458432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572847.446704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495458432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572847.446709:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572847.446711:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572847.446714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572847.446718:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458560:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572847.446729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572847.446743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff93a258da0380 x1875453495458560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572847.446751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495458560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572847.446756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572847.446759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572847.446761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458560:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572847.446764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495458688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572847.446776:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572847.446782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff93a258da1880 x1875453495458688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572847.446789:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495458688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572847.446794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572847.446796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572847.446798:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495458688:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572848.406414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572849.430383:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572850.454399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572851.478429:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572852.502383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572852.566426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.566442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572852.566448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.566464:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1069 00000100:00080000:0.0:1788572852.566478:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.566487:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572852.566540:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495458816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.566571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.566574:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572852.566576:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.566580:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1069 00000100:00080000:0.0:1788572852.566583:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.566586:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572852.566615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572852.566631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495458816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572852.566632:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495458944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.566647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.566650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572852.566651:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.566655:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1069 00000100:00080000:0.0:1788572852.566659:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.566662:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572852.566666:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572852.566675:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495459072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.566687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.566691:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572852.566693:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.566698:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1069 00000100:00080000:0.0:1788572852.566703:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.566707:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572852.566714:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572852.566714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788572852.566731:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495459200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572852.566773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff93a2460d1f80 x1875453495458816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572852.566791:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495458816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572852.566809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572852.566813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572852.566817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495458816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572852.566843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572852.566862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572852.566871:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff93a2460d2680 x1875453495458944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572852.566880:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495458944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572852.566885:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572852.566888:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572852.566890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495458944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572852.566894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459072:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572852.566906:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572852.566912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff93a2460d2a00 x1875453495459072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572852.566919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495459072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572852.566925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572852.566926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572852.566928:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459072:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572852.566937:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495459200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572852.566952:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572852.566959:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff93a2460d2300 x1875453495459200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572852.566967:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495459200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572852.566972:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572852.566973:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572852.566975:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495459200:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572853.526404:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572854.550340:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572855.574390:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572856.598402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572857.622458:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572857.686409:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572857.686424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572857.686429:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572857.686441:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1074 00000100:00080000:2.0:1788572857.686476:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572857.686484:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572857.686522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495459328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572857.686551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572857.686554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572857.686556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572857.686560:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1074 00000100:00080000:2.0:1788572857.686562:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572857.686565:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572857.686575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572857.686587:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495459456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572857.686596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572857.686600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572857.686602:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572857.686602:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495459328:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572857.686609:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1074 00000100:00080000:2.0:1788572857.686613:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572857.686615:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572857.686619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572857.686628:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495459584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572857.686636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572857.686644:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572857.686645:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572857.686648:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1074 00000100:00080000:2.0:1788572857.686662:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572857.686665:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572857.686669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572857.686678:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572857.686680:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495459712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572857.686746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff93a249409880 x1875453495459328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572857.686773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495459328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572857.686786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572857.686791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572857.686796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495459328:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572857.686826:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572857.686851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572857.686864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff93a249408700 x1875453495459456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572857.686890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495459456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572857.686899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572857.686903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572857.686907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572857.686914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459584:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572857.686940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572857.686955:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff93a24940aa00 x1875453495459584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572857.686969:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495459584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572857.686978:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572857.686982:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572857.686985:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459584:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572857.686993:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495459712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572857.687016:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572857.687029:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff93a249408a80 x1875453495459712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572857.687043:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495459712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572857.687065:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572857.687069:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572857.687073:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495459712:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572858.646413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572859.670384:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572860.694377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572861.718420:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572862.742582:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572862.806317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572862.806325:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572862.806329:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572862.806337:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1079 00000100:00080000:1.0:1788572862.806345:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572862.806352:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572862.806385:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495459840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572862.806414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572862.806419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572862.806421:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572862.806425:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1079 00000100:00080000:1.0:1788572862.806430:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572862.806435:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572862.806457:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495459840:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788572862.806461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572862.806481:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495459968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572862.806496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572862.806499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572862.806502:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572862.806507:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1079 00000100:00080000:1.0:1788572862.806512:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572862.806517:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572862.806524:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572862.806539:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495460096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572862.806551:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459968:192.168.202.102@tcp:38:kworker.0 00000100:00080000:1.0:1788572862.806556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572862.806561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572862.806565:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572862.806570:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1079 00000100:00080000:1.0:1788572862.806579:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572862.806585:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572862.806588:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460096:192.168.202.102@tcp:8:kworker.0 00000008:00100000:1.0:1788572862.806602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572862.806622:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495460224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572862.806660:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495460224:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572863.767169:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572864.791013:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572865.815664:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788572865.878509:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788572865.878548:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572865.878561:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572865.878601:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572865.878608:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788572865.878615:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572865.878662:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572865] req@ffff93a258da2680 x1875453495460224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572865.878691:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495460224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572865.878705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572865.878710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572865.878715:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495460224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572865.878723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572865] req@ffff93a258da1c00 x1875453495460096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572865.878735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495460096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572865.878742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572865.878765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572865.878769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460096:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572865.878777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572865] req@ffff93a258da0700 x1875453495459968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572865.878792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495459968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572865.878806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572865.878810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572865.878814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495459968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572865.878823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572865] req@ffff93a258da0a80 x1875453495459840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572865.878836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495459840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572865.878844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572865.878847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572865.878850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495459840:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788572866.839176:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572867.862476:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572867.926413:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572867.926433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572867.926436:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572867.926444:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1084 00000100:00080000:0.0:1788572867.926453:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572867.926459:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572867.926502:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495460352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572867.926540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572867.926545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572867.926547:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572867.926552:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1084 00000100:00080000:0.0:1788572867.926556:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572867.926561:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572867.926576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572867.926591:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495460480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572867.926603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572867.926607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572867.926609:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572867.926614:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1084 00000100:00080000:0.0:1788572867.926618:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572867.926624:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495460352:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572867.926626:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572867.926636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572867.926650:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495460608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572867.926670:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572867.926674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572867.926677:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572867.926681:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1084 00000100:00080000:0.0:1788572867.926687:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572867.926693:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572867.926699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572867.926708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788572867.926716:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495460736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572867.926764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff93a2460d1180 x1875453495460352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572867.926787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495460352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572867.926811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572867.926816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572867.926821:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495460352:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572867.926829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572867.926857:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572867.926872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff93a2460d1500 x1875453495460480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572867.926886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495460480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572867.926895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572867.926898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572867.926907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572867.926942:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460608:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572867.926967:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572867.926979:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff93a2460d3480 x1875453495460608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572867.926991:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495460608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572867.927000:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572867.927003:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572867.927007:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460608:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572867.927021:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495460736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572867.927045:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572867.927058:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff93a2460d0380 x1875453495460736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572867.927070:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495460736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572867.927079:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572867.927082:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572867.927086:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495460736:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572868.886407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572869.910361:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572870.934404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572871.958398:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572872.982393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572873.047275:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572873.047284:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572873.047287:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572873.047296:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1089 00000100:00080000:0.0:1788572873.047304:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572873.047311:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572873.047350:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495460864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572873.047384:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572873.047389:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572873.047391:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572873.047396:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1089 00000100:00080000:0.0:1788572873.047400:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572873.047405:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572873.047430:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572873.047436:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495460864:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572873.047449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495460992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572873.047459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572873.047462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572873.047463:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572873.047467:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1089 00000100:00080000:0.0:1788572873.047471:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572873.047506:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572873.047514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572873.047519:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788572873.047534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495461120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572873.047554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572873.047557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572873.047558:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572873.047562:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1089 00000100:00080000:0.0:1788572873.047566:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572873.047567:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff93a2460d3b80 x1875453495460864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572873.047572:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572873.047579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572873.047591:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495460864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572873.047592:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495461248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572873.047605:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572873.047612:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572873.047617:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495460864:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572873.047637:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572873.047655:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572873.047664:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff93a2460d0a80 x1875453495460992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572873.047680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495460992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572873.047686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572873.047688:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572873.047690:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495460992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572873.047695:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495461120:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572873.047709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572873.047716:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff93a2460d0000 x1875453495461120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572873.047723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495461120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572873.047728:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572873.047739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572873.047746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495461120:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572873.047752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495461248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572873.047773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572873.047785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff93a2460d1c00 x1875453495461248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572873.047797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495461248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572873.047818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572873.047822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572873.047824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495461248:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572874.006353:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572875.030417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572876.054360:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572877.078419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572878.102463:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572878.166432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572878.166443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572878.166448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572878.166459:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1094 00000100:00080000:3.0:1788572878.166470:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572878.166478:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572878.166516:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495461376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572878.166552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572878.166556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572878.166558:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572878.166563:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1094 00000100:00080000:3.0:1788572878.166580:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572878.166586:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572878.166596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572878.166611:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495461504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572878.166625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572878.166631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788572878.166635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495461376:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572878.166648:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572878.166655:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1094 00000100:00080000:3.0:1788572878.166660:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572878.166665:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572878.166673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572878.166692:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495461632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572878.166705:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572878.166709:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572878.166713:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572878.166717:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1094 00000100:00080000:3.0:1788572878.166722:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572878.166727:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:2.0:1788572878.166735:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788572878.166737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572878.166754:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495461760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572878.166806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff93a25064e300 x1875453495461376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572878.166826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495461376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572878.166836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572878.166840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572878.166843:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495461376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572878.166868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495461504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572878.166889:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572878.166898:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff93a25064c380 x1875453495461504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572878.166911:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495461504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572878.166919:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572878.166923:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572878.166926:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495461504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572878.166949:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495461632:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572878.166993:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572878.167009:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff93a25064f480 x1875453495461632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572878.167022:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495461632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572878.167031:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572878.167034:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572878.167038:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495461632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572878.167045:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495461760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572878.167073:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572878.167088:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff93a25064ce00 x1875453495461760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572878.167099:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495461760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572878.167107:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572878.167111:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572878.167115:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495461760:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572879.126433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572880.150478:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572881.174487:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572882.198455:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572883.222442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572883.286360:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572883.286372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572883.286399:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572883.286413:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1099 00000100:00080000:0.0:1788572883.286424:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572883.286431:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572883.286467:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495461888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572883.286496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572883.286499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572883.286500:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572883.286504:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1099 00000100:00080000:0.0:1788572883.286506:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572883.286508:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572883.286518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572883.286528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495462016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572883.286534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572883.286537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572883.286539:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572883.286542:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1099 00000100:00080000:0.0:1788572883.286545:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572883.286551:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572883.286553:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495461888:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788572883.286557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572883.286572:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495462144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572883.286592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572883.286596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572883.286597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572883.286600:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1099 00000100:00080000:0.0:1788572883.286604:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572883.286607:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572883.286611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572883.286634:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495462272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572883.286649:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572883.286714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff93a2460d0700 x1875453495461888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.286736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495461888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.286748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.286753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572883.286758:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495461888:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572883.286766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572883.286793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572883.286806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff93a2460d1c00 x1875453495462016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.286819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495462016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.286826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.286830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572883.286833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572883.286861:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462144:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572883.286882:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572883.286892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff93a2460d0000 x1875453495462144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.286913:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495462144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.286921:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.286924:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572883.286927:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462144:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572883.286934:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495462272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572883.286954:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572883.286965:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff93a2460d0a80 x1875453495462272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.286977:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495462272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.286985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.286988:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572883.286991:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495462272:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572884.246371:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572885.270463:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572886.294420:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572887.318447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572888.342424:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572888.406380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.406391:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572888.406396:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.406408:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1104 00000100:00080000:0.0:1788572888.406421:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.406431:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572888.406477:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495462400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572888.406506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.406509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572888.406511:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.406515:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1104 00000100:00080000:0.0:1788572888.406519:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.406523:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572888.406554:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495462400:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788572888.406556:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572888.406571:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495462528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572888.406582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.406586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572888.406588:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.406593:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1104 00000100:00080000:0.0:1788572888.406599:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.406603:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572888.406609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572888.406620:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495462656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572888.406624:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572888.406635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.406639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572888.406641:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.406646:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1104 00000100:00080000:0.0:1788572888.406650:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.406655:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572888.406660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572888.406669:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff93a2460d3b80 x1875453495462400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.406672:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495462784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572888.406709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495462400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.406718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.406722:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572888.406729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495462400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572888.406751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572888.406768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572888.406780:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff93a2460d0380 x1875453495462528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.406792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495462528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.406807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.406811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572888.406814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572888.406819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462656:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572888.406843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572888.406851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff93a2460d3480 x1875453495462656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.406858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495462656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.406871:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.406873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572888.406875:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495462656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572888.406879:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495462784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572888.406892:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572888.406898:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff93a2460d1500 x1875453495462784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.406905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495462784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.406909:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.406911:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572888.406912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495462784:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572889.366378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572890.390392:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572891.414349:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572892.438369:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572893.462401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572893.526371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572893.526387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572893.526392:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572893.526403:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1109 00000100:00080000:2.0:1788572893.526414:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572893.526423:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572893.526490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495462912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572893.526524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572893.526528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572893.526530:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572893.526535:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1109 00000100:00080000:2.0:1788572893.526539:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572893.526545:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572893.526557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572893.526567:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495462912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572893.526571:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495463040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572893.526581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572893.526583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572893.526585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572893.526587:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1109 00000100:00080000:2.0:1788572893.526590:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572893.526592:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572893.526595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572893.526605:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495463168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572893.526616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572893.526619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572893.526621:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572893.526635:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1109 00000100:00080000:2.0:1788572893.526640:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572893.526644:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572893.526650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572893.526658:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572893.526664:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495463296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572893.526721:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff93a249408e00 x1875453495462912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.526770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495462912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.526782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.526792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572893.526797:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495462912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572893.526825:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572893.526850:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572893.526863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff93a249408000 x1875453495463040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.526875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495463040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.526884:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.526887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572893.526890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572893.526896:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463168:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572893.526916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572893.526927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff93a24940ad80 x1875453495463168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.526938:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495463168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.526945:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.526948:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572893.526951:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463168:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572893.526957:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495463296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572893.526981:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572893.526992:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff93a249409180 x1875453495463296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.527001:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495463296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.527008:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.527011:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572893.527014:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495463296:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572894.486342:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572895.510423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572896.534401:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572897.558423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572898.582437:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572898.646355:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572898.646364:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572898.646368:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572898.646377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1115 00000100:00080000:3.0:1788572898.646388:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572898.646394:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572898.646439:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495463424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572898.646463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572898.646466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572898.646468:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572898.646471:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1115 00000100:00080000:3.0:1788572898.646473:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572898.646476:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572898.646484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572898.646509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495463552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572898.646523:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495463424:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788572898.646526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572898.646535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572898.646543:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572898.646554:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1115 00000100:00080000:3.0:1788572898.646558:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572898.646562:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572898.646568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572898.646576:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495463680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572898.646584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572898.646587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572898.646588:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572898.646591:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1115 00000100:00080000:3.0:1788572898.646606:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572898.646609:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572898.646613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572898.646620:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495463808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572898.646625:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572898.646679:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff93a25064f100 x1875453495463424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.646697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495463424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.646707:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.646711:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572898.646714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495463424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572898.646740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572898.646758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572898.646767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff93a25064d500 x1875453495463552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.646807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495463552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.646813:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.646816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572898.646818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572898.646823:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463680:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572898.646842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572898.646850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff93a25064c700 x1875453495463680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.646858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495463680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.646870:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.646873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572898.646875:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495463680:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572898.646880:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495463808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572898.646894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572898.646902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff93a25064ca80 x1875453495463808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.646909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495463808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.646924:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.646927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572898.646929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495463808:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572899.606418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572900.630448:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572901.654438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572902.678384:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572903.702382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572903.766350:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.766359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572903.766363:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.766371:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1120 00000100:00080000:2.0:1788572903.766379:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.766385:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572903.766416:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495463936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572903.766438:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.766441:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572903.766443:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.766446:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1120 00000100:00080000:2.0:1788572903.766448:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.766451:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572903.766460:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572903.766468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495464064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572903.766480:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495463936:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572903.766495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.766498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572903.766499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.766504:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1120 00000100:00080000:2.0:1788572903.766507:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.766510:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572903.766515:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572903.766529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495464192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572903.766540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.766543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572903.766545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788572903.766551:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572903.766552:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1120 00000100:00080000:2.0:1788572903.766560:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.766565:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572903.766578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572903.766590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495464320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572903.766610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff93a24940a680 x1875453495463936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572903.766641:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495463936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572903.766653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572903.766658:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572903.766662:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495463936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572903.766684:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572903.766710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572903.766724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff93a249409c00 x1875453495464064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572903.766737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495464064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572903.766744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572903.766746:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572903.766748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572903.766753:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464192:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572903.766769:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572903.766777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff93a24940b800 x1875453495464192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572903.766784:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495464192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572903.766798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572903.766800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572903.766802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572903.766806:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495464320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572903.766820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572903.766827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff93a249409180 x1875453495464320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572903.766834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495464320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572903.766839:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572903.766840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572903.766842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495464320:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572904.726388:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572905.750374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572906.774392:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572907.798433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572908.822418:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572908.886392:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.886404:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572908.886409:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.886420:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1125 00000100:00080000:2.0:1788572908.886441:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.886450:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572908.886502:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495464448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.886533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.886536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572908.886538:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.886542:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1125 00000100:00080000:2.0:1788572908.886545:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.886548:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572908.886558:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572908.886569:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495464576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.886576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.886579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572908.886580:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.886583:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1125 00000100:00080000:2.0:1788572908.886584:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.886586:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572908.886589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572908.886616:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495464704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.886624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.886627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572908.886628:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.886630:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1125 00000100:00080000:2.0:1788572908.886633:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.886636:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572908.886638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572908.886649:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495464832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572908.886656:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495464448:192.168.202.102@tcp:250:kworker.0 00000800:00000100:3.0:1788572908.886746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572908.886786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff93a24940ad80 x1875453495464448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572908.886801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495464448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572908.886810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572908.886814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572908.886817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495464448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572908.886839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572908.886858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572908.886866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff93a249408000 x1875453495464576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572908.886874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495464576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572908.886880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572908.886882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572908.886884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572908.886888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464704:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572908.886899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572908.886922:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff93a249408e00 x1875453495464704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572908.886930:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495464704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572908.886935:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572908.886938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572908.886940:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495464704:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572908.886960:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495464832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572908.886974:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572908.886981:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff93a249408a80 x1875453495464832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572908.886996:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495464832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572908.887005:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572908.887008:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572908.887011:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495464832:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572909.846434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572910.870372:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572911.894393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572912.918359:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572913.942369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572914.006346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.006355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572914.006372:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.006381:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1130 00000100:00080000:1.0:1788572914.006389:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.006395:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572914.006422:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495464960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572914.006444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.006446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572914.006448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.006450:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1130 00000100:00080000:1.0:1788572914.006452:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.006463:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572914.006474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572914.006489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495465088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572914.006495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495464960:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572914.006512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.006515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572914.006516:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.006521:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1130 00000100:00080000:1.0:1788572914.006524:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.006527:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572914.006532:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572914.006544:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495465216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572914.006569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572914.006570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788572914.006576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572914.006579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.006585:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1130 00000100:00080000:1.0:1788572914.006591:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.006596:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572914.006601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572914.006612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495465344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572914.006635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff93a258da1f80 x1875453495464960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572914.006657:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495464960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572914.006668:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572914.006672:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572914.006677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495464960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572914.006700:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572914.006725:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572914.006738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff93a258da0e00 x1875453495465088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572914.006752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495465088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572914.006775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572914.006780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572914.006784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572914.006790:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465216:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572914.006813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572914.006824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff93a258da3800 x1875453495465216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572914.006836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495465216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572914.006845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572914.006849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572914.006852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572914.006859:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495465344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572914.006880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572914.006892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff93a258da0a80 x1875453495465344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572914.006904:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495465344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572914.006912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572914.006915:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572914.006926:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495465344:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572914.966365:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572915.990396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572917.014361:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572918.038454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572919.062445:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572919.126369:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.126384:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572919.126390:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.126404:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1135 00000100:00080000:3.0:1788572919.126418:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.126428:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572919.126489:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495465472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572919.126524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.126534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572919.126536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.126551:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1135 00000100:00080000:3.0:1788572919.126555:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.126560:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572919.126571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572919.126583:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495465472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572919.126589:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495465600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572919.126600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.126602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572919.126604:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.126608:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1135 00000100:00080000:3.0:1788572919.126611:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.126632:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572919.126638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572919.126647:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495465728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572919.126656:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.126658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572919.126660:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.126662:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1135 00000100:00080000:3.0:1788572919.126666:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788572919.126668:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788572919.126672:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572919.126677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572919.126686:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495465856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572919.126727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff93a25064df80 x1875453495465472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572919.126746:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495465472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572919.126762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572919.126766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572919.126770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495465472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572919.126795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572919.126814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572919.126823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff93a25064ca80 x1875453495465600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572919.126831:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495465600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572919.126837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572919.126839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572919.126842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572919.126845:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465728:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572919.126857:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572919.126862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff93a25064c700 x1875453495465728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572919.126870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495465728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572919.126875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572919.126877:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572919.126879:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495465728:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572919.126887:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495465856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572919.126901:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572919.126907:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff93a25064d500 x1875453495465856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572919.126915:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495465856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572919.126920:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572919.126922:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572919.126924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495465856:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572920.086458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572921.110417:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572922.134445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572923.158425:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572924.182465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572924.246346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.246359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572924.246363:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.246374:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1140 00000100:00080000:3.0:1788572924.246384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.246392:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572924.246424:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495465984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572924.246454:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.246458:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572924.246474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.246481:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1140 00000100:00080000:3.0:1788572924.246486:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.246491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572924.246497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495465984:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788572924.246504:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572924.246523:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495466112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572924.246537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.246541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572924.246543:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.246548:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1140 00000800:00000100:2.0:1788572924.246554:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788572924.246554:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.246563:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788572924.246569:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572924.246584:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495466240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572924.246613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.246617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1788572924.246622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff93a25064f100 x1875453495465984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572924.246622:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.246630:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1140 00000100:00080000:3.0:1788572924.246647:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572924.246650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495465984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572924.246656:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:2.0:1788572924.246664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572924.246666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788572924.246670:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572924.246673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495465984:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572924.246690:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495466368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572924.246714:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572924.246736:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572924.246745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff93a25064ce00 x1875453495466112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572924.246755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495466112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572924.246760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572924.246763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572924.246765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572924.246769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466240:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572924.246781:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572924.246788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff93a25064f480 x1875453495466240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572924.246813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495466240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572924.246823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572924.246826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572924.246830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466240:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572924.246837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495466368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572924.246860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572924.246871:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff93a25064c380 x1875453495466368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572924.246886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495466368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572924.246892:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572924.246894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572924.246896:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495466368:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572925.206483:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572926.230430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572927.254441:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572928.278393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572929.302896:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572929.366353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572929.366368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572929.366394:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572929.366407:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1145 00000100:00080000:0.0:1788572929.366422:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572929.366431:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572929.366480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495466496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572929.366516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572929.366521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572929.366523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572929.366528:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1145 00000100:00080000:0.0:1788572929.366532:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572929.366537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572929.366549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572929.366567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495466624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572929.366579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572929.366584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572929.366586:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572929.366592:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1145 00000100:00080000:0.0:1788572929.366596:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572929.366605:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495466496:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572929.366607:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572929.366626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572929.366647:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495466752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572929.366678:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572929.366683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572929.366685:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572929.366691:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1145 00000100:00080000:0.0:1788572929.366697:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572929.366721:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572929.366754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466752:192.168.202.102@tcp:8:kworker.0 00000100:00080000:0.0:1788572929.366767:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572929.366783:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572929.366813:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495466880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572929.366868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495466880:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572930.326408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572931.350878:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572932.374934:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788572932.438497:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788572932.438542:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572932.438564:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788572932.438609:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788572932.438617:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788572932.438625:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:2.0:1788572932.438663:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff93a2460d2680 x1875453495466880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572932.438697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495466880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572932.438711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572932.438716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572932.438739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495466880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572932.438747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff93a2460d2a00 x1875453495466752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572932.438760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495466752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572932.438769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572932.438774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572932.438778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572932.438786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff93a2460d1180 x1875453495466496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572932.438800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495466496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572932.438808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572932.438812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572932.438829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495466496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572932.438838:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff93a2460d2300 x1875453495466624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572932.438853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495466624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572932.438862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572932.438866:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572932.438887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495466624:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572933.399169:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572934.422467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572934.486332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572934.486343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572934.486348:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572934.486360:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1150 00000100:00080000:1.0:1788572934.486375:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572934.486387:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572934.486439:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495467008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572934.486476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572934.486481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572934.486483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572934.486489:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1150 00000100:00080000:1.0:1788572934.486494:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572934.486501:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572934.486533:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495467008:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788572934.486537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572934.486578:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495467136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572934.486592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572934.486596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572934.486597:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572934.486604:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1150 00000100:00080000:1.0:1788572934.486610:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572934.486615:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572934.486625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572934.486625:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572934.486647:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495467264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572934.486661:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572934.486667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572934.486670:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572934.486678:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1150 00000100:00080000:1.0:1788572934.486683:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572934.486691:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572934.486693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff93a258da0700 x1875453495467008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788572934.486698:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572934.486717:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495467392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572934.486733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495467008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572934.486769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572934.486777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572934.486785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495467008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572934.486829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572934.486870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572934.486884:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff93a258da1c00 x1875453495467136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572934.486897:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495467136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572934.486905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572934.486909:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572934.486913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572934.486919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467264:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572934.486941:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572934.486971:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff93a258da2680 x1875453495467264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572934.486983:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495467264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572934.486991:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572934.486995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572934.486998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467264:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572934.487006:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495467392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572934.487036:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572934.487048:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff93a258da1880 x1875453495467392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572934.487059:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495467392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572934.487066:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572934.487069:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572934.487072:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495467392:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572935.446404:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572936.470398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572937.494402:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572938.518392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572939.542442:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572939.606349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.606357:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572939.606385:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.606394:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1156 00000100:00080000:1.0:1788572939.606402:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.606408:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572939.606440:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495467520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572939.606463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.606465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572939.606467:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.606470:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1156 00000100:00080000:1.0:1788572939.606472:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.606474:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572939.606482:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572939.606490:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495467648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572939.606498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.606501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572939.606502:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.606505:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1156 00000100:00080000:1.0:1788572939.606507:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.606509:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572939.606513:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572939.606515:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495467520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788572939.606522:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495467776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572939.606541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.606543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572939.606545:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.606548:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1156 00000100:00080000:1.0:1788572939.606551:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.606554:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572939.606557:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572939.606565:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495467904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572939.606582:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572939.606629:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff93a258da0380 x1875453495467520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572939.606648:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495467520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572939.606666:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572939.606671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572939.606674:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495467520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572939.606680:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572939.606700:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572939.606710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff93a258da2a00 x1875453495467648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572939.606718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495467648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572939.606724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572939.606726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572939.606728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572939.606746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467776:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572939.606758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572939.606765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff93a258da3480 x1875453495467776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572939.606772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495467776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572939.606777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572939.606779:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572939.606781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495467776:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572939.606791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495467904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572939.606804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572939.606811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff93a258da3100 x1875453495467904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572939.606819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495467904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572939.606835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572939.606837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572939.606839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495467904:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572940.566389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572941.590432:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572942.614414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572943.638400:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572944.662392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572944.726400:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572944.726409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572944.726412:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572944.726438:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1161 00000100:00080000:0.0:1788572944.726447:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572944.726453:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572944.726483:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495468032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572944.726511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572944.726515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572944.726517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572944.726523:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1161 00000100:00080000:0.0:1788572944.726527:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572944.726532:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572944.726542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572944.726560:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495468160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572944.726568:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495468032:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788572944.726576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572944.726581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572944.726583:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572944.726589:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1161 00000100:00080000:0.0:1788572944.726593:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572944.726597:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572944.726603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572944.726617:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495468288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572944.726628:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572944.726644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572944.726648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572944.726650:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572944.726655:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1161 00000100:00080000:0.0:1788572944.726661:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572944.726666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff93a2460d1f80 x1875453495468032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572944.726668:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572944.726677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572944.726687:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495468032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572944.726693:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495468416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572944.726699:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572944.726706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572944.726711:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495468032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572944.726730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572944.726746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572944.726754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff93a2460d2300 x1875453495468160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572944.726770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495468160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572944.726776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572944.726778:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572944.726780:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572944.726792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468288:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572944.726807:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572944.726814:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff93a2460d1180 x1875453495468288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572944.726821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495468288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572944.726826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572944.726828:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572944.726830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572944.726833:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495468416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572944.726842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572944.726848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff93a2460d2a00 x1875453495468416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572944.726855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495468416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572944.726869:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572944.726871:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572944.726873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495468416:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572945.686439:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572946.710459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572947.734406:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572948.758448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572949.782420:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572949.846384:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.846395:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572949.846400:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.846412:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1166 00000100:00080000:1.0:1788572949.846424:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.846431:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572949.846474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495468544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.846502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.846506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572949.846507:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.846510:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1166 00000100:00080000:1.0:1788572949.846513:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.846515:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572949.846543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572949.846555:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495468544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788572949.846556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495468672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.846571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.846573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572949.846575:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.846578:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1166 00000100:00080000:1.0:1788572949.846581:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.846584:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572949.846587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572949.846595:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495468800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.846603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.846607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572949.846608:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.846611:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1166 00000100:00080000:1.0:1788572949.846613:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.846615:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572949.846618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572949.846627:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495468928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572949.846639:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572949.846685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff93a258da1180 x1875453495468544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572949.846700:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495468544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572949.846709:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572949.846712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572949.846716:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495468544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572949.846741:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572949.846764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572949.846779:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff93a258da0000 x1875453495468672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572949.846793:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495468672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572949.846802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572949.846805:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572949.846808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788572949.846820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468800:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788572949.846839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572949.846847:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff93a258da2300 x1875453495468800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572949.846855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495468800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572949.846861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572949.846863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572949.846866:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495468800:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788572949.846875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495468928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788572949.846887:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572949.846894:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff93a258da3100 x1875453495468928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572949.846901:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495468928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572949.846906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572949.846908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788572949.846910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495468928:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572950.806424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572951.830458:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572952.854491:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572953.878371:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572954.902415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572954.966323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572954.966332:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572954.966336:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572954.966361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1171 00000100:00080000:2.0:1788572954.966370:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572954.966377:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572954.966408:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495469056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572954.966432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572954.966435:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572954.966436:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572954.966440:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1171 00000100:00080000:2.0:1788572954.966444:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572954.966448:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572954.966459:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572954.966473:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495469056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572954.966476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495469184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572954.966490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572954.966494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572954.966496:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572954.966501:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1171 00000100:00080000:2.0:1788572954.966505:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572954.966510:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572954.966516:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572954.966530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495469312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572954.966539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788572954.966551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572954.966554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572954.966556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572954.966560:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1171 00000100:00080000:2.0:1788572954.966564:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572954.966567:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572954.966571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572954.966580:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495469440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572954.966585:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff93a24940aa00 x1875453495469056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572954.966604:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495469056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572954.966613:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572954.966617:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572954.966620:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495469056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572954.966651:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572954.966667:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572954.966683:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff93a249408700 x1875453495469184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572954.966692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495469184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572954.966709:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572954.966711:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572954.966714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572954.966719:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469312:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572954.966746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572954.966753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff93a249409880 x1875453495469312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572954.966761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495469312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572954.966766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572954.966768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572954.966770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469312:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572954.966774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495469440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572954.966784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572954.966797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff93a249409500 x1875453495469440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572954.966805:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495469440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572954.966810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572954.966812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572954.966820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495469440:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572955.926361:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572956.950408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572957.974353:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572958.998416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572960.022361:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572960.086317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572960.086326:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572960.086330:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572960.086339:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1176 00000100:00080000:2.0:1788572960.086353:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572960.086360:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572960.086392:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495469568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572960.086411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572960.086414:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572960.086415:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572960.086418:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1176 00000100:00080000:2.0:1788572960.086420:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572960.086423:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572960.086444:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572960.086455:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495469568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572960.086457:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495469696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572960.086472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572960.086474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572960.086475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572960.086478:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1176 00000100:00080000:2.0:1788572960.086481:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572960.086483:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572960.086487:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572960.086494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495469824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572960.086501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572960.086504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572960.086506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572960.086508:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1176 00000100:00080000:2.0:1788572960.086511:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572960.086513:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572960.086516:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572960.086526:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788572960.086527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495469952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572960.086572:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff93a24940b100 x1875453495469568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572960.086587:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495469568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572960.086596:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572960.086600:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572960.086603:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495469568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572960.086622:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572960.086637:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572960.086645:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff93a24940bb80 x1875453495469696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572960.086652:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495469696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572960.086657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572960.086659:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572960.086661:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572960.086665:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469824:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572960.086676:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572960.086691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff93a24940b480 x1875453495469824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572960.086699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495469824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572960.086705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572960.086707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572960.086709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495469824:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572960.086712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495469952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572960.086724:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572960.086730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff93a249409500 x1875453495469952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572960.086737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495469952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572960.086742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572960.086744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572960.086745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495469952:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572961.046419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572962.070364:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572963.094475:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572964.118462:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572965.142456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572965.206393:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572965.206409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572965.206437:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572965.206448:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1181 00000100:00080000:1.0:1788572965.206459:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572965.206467:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572965.206510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495470080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572965.206538:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572965.206541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572965.206543:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572965.206546:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1181 00000100:00080000:1.0:1788572965.206549:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572965.206551:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572965.206562:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572965.206572:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495470208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572965.206580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572965.206583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572965.206584:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572965.206588:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1181 00000100:00080000:1.0:1788572965.206589:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572965.206594:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788572965.206602:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495470080:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788572965.206603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572965.206616:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495470336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572965.206639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572965.206642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572965.206644:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572965.206649:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1181 00000100:00080000:1.0:1788572965.206653:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572965.206656:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572965.206671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572965.206675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788572965.206700:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495470464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572965.206742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff93a258da3480 x1875453495470080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572965.206762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495470080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572965.206785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572965.206789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572965.206792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495470080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572965.206801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572965.206827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572965.206854:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff93a258da2a00 x1875453495470208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572965.206865:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495470208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572965.206870:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572965.206873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572965.206875:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572965.206896:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470336:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572965.206913:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572965.206921:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff93a258da0380 x1875453495470336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572965.206933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495470336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572965.206941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572965.206944:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572965.206948:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470336:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572965.206962:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495470464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572965.206982:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572965.206991:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff93a258da1880 x1875453495470464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572965.206999:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495470464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572965.207004:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572965.207006:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572965.207008:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495470464:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572966.166423:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572967.190392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572968.214467:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572969.238402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572970.262463:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572970.326350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572970.326363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572970.326368:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572970.326379:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1186 00000100:00080000:1.0:1788572970.326392:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572970.326398:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572970.326441:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495470592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572970.326469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572970.326472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572970.326473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572970.326477:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1186 00000100:00080000:1.0:1788572970.326479:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572970.326482:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572970.326511:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572970.326521:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495470720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572970.326535:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495470592:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572970.326537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572970.326557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572970.326560:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572970.326568:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1186 00000100:00080000:1.0:1788572970.326574:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572970.326580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572970.326588:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572970.326603:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495470848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572970.326613:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788572970.326627:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572970.326631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572970.326634:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572970.326640:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1186 00000100:00080000:1.0:1788572970.326644:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572970.326647:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572970.326654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572970.326671:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495470976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572970.326681:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff93a258da2680 x1875453495470592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572970.326709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495470592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572970.326723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572970.326727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572970.326730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495470592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572970.326751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572970.326771:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572970.326781:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff93a258da1c00 x1875453495470720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572970.326802:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495470720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572970.326808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572970.326811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572970.326813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572970.326817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470848:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572970.326833:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572970.326840:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff93a258da0700 x1875453495470848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572970.326847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495470848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572970.326852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572970.326854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572970.326856:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495470848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572970.326860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495470976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572970.326870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572970.326883:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff93a258da0a80 x1875453495470976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572970.326896:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495470976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572970.326917:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572970.326921:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572970.326925:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495470976:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572971.286368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572972.310369:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572973.334391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572974.358358:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572975.382475:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572975.446362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572975.446377:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572975.446408:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572975.446423:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1191 00000100:00080000:1.0:1788572975.446439:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572975.446448:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788572975.446501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495471104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572975.446537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572975.446542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572975.446545:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572975.446550:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1191 00000100:00080000:1.0:1788572975.446554:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572975.446559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572975.446572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572975.446587:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495471232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572975.446605:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495471104:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788572975.446609:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572975.446615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572975.446617:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572975.446623:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1191 00000100:00080000:1.0:1788572975.446628:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572975.446633:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572975.446639:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572975.446653:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495471360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572975.446680:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572975.446685:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788572975.446688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572975.446692:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572975.446698:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1191 00000100:00080000:1.0:1788572975.446704:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572975.446710:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788572975.446717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572975.446744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff93a258da3800 x1875453495471104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572975.446746:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495471488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572975.446783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495471104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572975.446798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572975.446803:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572975.446809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495471104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572975.446839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572975.446864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572975.446879:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff93a258da0e00 x1875453495471232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572975.446891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495471232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572975.446900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572975.446903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572975.446906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572975.446923:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471360:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572975.446947:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572975.446959:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff93a258da1f80 x1875453495471360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572975.446967:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495471360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572975.446973:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572975.446975:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572975.446978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471360:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572975.446981:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495471488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572975.446994:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572975.447000:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff93a258da0a80 x1875453495471488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572975.447007:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495471488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572975.447012:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572975.447014:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572975.447016:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495471488:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572976.406457:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572977.430399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572978.454432:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572979.478373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572980.502453:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572980.566390:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572980.566401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572980.566421:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572980.566429:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1197 00000100:00080000:2.0:1788572980.566438:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572980.566443:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572980.566475:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495471616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572980.566502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572980.566505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572980.566506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572980.566509:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1197 00000100:00080000:2.0:1788572980.566511:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572980.566513:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572980.566522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572980.566530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495471744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572980.566537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572980.566540:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572980.566542:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572980.566544:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1197 00000100:00080000:2.0:1788572980.566546:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572980.566550:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572980.566553:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572980.566560:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495471872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572980.566570:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495471616:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788572980.566596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572980.566600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572980.566601:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572980.566606:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1197 00000100:00080000:2.0:1788572980.566609:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572980.566612:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572980.566617:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572980.566627:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495472000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572980.566659:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572980.566707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff93a249409880 x1875453495471616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572980.566738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495471616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572980.566748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572980.566752:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572980.566755:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495471616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572980.566762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572980.566781:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572980.566789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff93a249408700 x1875453495471744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572980.566798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495471744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572980.566803:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572980.566805:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572980.566807:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572980.566831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471872:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572980.566843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572980.566850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff93a24940aa00 x1875453495471872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572980.566858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495471872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572980.566872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572980.566874:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572980.566877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495471872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572980.566881:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495472000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572980.566894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572980.566901:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff93a249408a80 x1875453495472000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572980.566909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495472000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572980.566914:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572980.566916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572980.566918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495472000:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572981.526378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572982.550401:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572983.574374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572984.598376:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572985.622402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572985.686362:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.686370:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572985.686374:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.686382:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1202 00000100:00080000:0.0:1788572985.686390:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.686396:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572985.686426:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495472128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572985.686463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.686473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572985.686475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.686479:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1202 00000100:00080000:0.0:1788572985.686482:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.686486:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572985.686494:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572985.686502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495472128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572985.686507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495472256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572985.686516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.686518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572985.686519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.686522:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1202 00000100:00080000:0.0:1788572985.686524:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.686526:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572985.686529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572985.686537:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495472384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572985.686543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.686546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572985.686547:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.686550:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1202 00000100:00080000:0.0:1788572985.686559:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572985.686559:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788572985.686567:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572985.686571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572985.686581:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495472512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572985.686602:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff93a2460d2680 x1875453495472128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572985.686617:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495472128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572985.686626:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572985.686629:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572985.686632:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495472128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572985.686657:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572985.686675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572985.686684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff93a2460d1500 x1875453495472256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572985.686692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495472256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572985.686697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572985.686700:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572985.686702:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572985.686706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472384:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572985.686717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572985.686723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff93a2460d3480 x1875453495472384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572985.686730:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495472384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572985.686735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572985.686737:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572985.686739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472384:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572985.686742:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495472512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572985.686766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572985.686773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff93a2460d0380 x1875453495472512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572985.686786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495472512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572985.686792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572985.686794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572985.686796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495472512:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572986.646447:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572987.670437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572988.694348:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572989.718403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572990.742396:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572990.806344:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572990.806356:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572990.806360:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572990.806372:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1207 00000100:00080000:0.0:1788572990.806383:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572990.806390:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788572990.806430:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495472640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572990.806460:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572990.806465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572990.806467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572990.806472:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1207 00000100:00080000:0.0:1788572990.806476:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572990.806481:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572990.806492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572990.806509:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495472640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788572990.806510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495472768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572990.806540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572990.806545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572990.806547:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572990.806553:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1207 00000100:00080000:0.0:1788572990.806562:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572990.806567:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788572990.806573:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788572990.806574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572990.806586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495472896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572990.806595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572990.806598:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572990.806600:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572990.806602:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1207 00000100:00080000:0.0:1788572990.806606:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572990.806608:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788572990.806612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572990.806620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff93a2460d3b80 x1875453495472640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572990.806621:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495473024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572990.806661:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495472640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572990.806673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572990.806677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572990.806682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495472640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788572990.806704:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572990.806727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572990.806740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff93a2460d0a80 x1875453495472768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572990.806753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495472768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572990.806776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572990.806780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572990.806783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572990.806789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472896:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788572990.806817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572990.806827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff93a2460d0000 x1875453495472896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572990.806838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495472896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572990.806845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572990.806848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572990.806851:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495472896:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572990.806856:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495473024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788572990.806871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788572990.806881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff93a2460d1c00 x1875453495473024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572990.806893:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495473024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572990.806900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572990.806903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572990.806911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495473024:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788572991.766482:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572992.790516:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572993.814446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572994.838457:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572995.862784:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572995.926418:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572995.926433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572995.926439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572995.926454:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1212 00000100:00080000:2.0:1788572995.926468:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572995.926478:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788572995.926534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495473152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572995.926573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572995.926577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572995.926579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572995.926604:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1212 00000100:00080000:2.0:1788572995.926609:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572995.926615:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572995.926629:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572995.926631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495473152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788572995.926648:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495473280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572995.926661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572995.926665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572995.926667:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572995.926672:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1212 00000100:00080000:2.0:1788572995.926676:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572995.926681:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572995.926687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572995.926700:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495473408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572995.926712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572995.926716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572995.926718:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572995.926723:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1212 00000100:00080000:2.0:1788572995.926727:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572995.926732:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788572995.926738:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572995.926752:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495473536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572995.926764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572995.926798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572995.926827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495473536:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788572996.886437:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572997.912137:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572998.934430:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788572998.998457:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788572998.998493:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572998.998521:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572998.998565:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572998.998577:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788572998.998586:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788572998.998643:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572998] req@ffff93a249409180 x1875453495473536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572998.998669:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495473536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572998.998681:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572998.998686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572998.998691:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495473536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572998.998699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572998] req@ffff93a249408000 x1875453495473280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572998.998711:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495473280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572998.998719:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572998.998723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572998.998726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788572998.998733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572998] req@ffff93a24940ad80 x1875453495473408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572998.998753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495473408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572998.998762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572998.998765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572998.998768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788572998.998775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572998] req@ffff93a249408e00 x1875453495473152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572998.998787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495473152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572998.998795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572998.998798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788572998.998802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495473152:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788572999.958876:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573000.982429:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573001.046332:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573001.046345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573001.046349:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573001.046360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1217 00000100:00080000:2.0:1788573001.046371:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573001.046379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573001.046413:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495473664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573001.046444:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573001.046448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573001.046450:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573001.046455:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1217 00000100:00080000:2.0:1788573001.046459:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573001.046463:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573001.046474:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573001.046490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495473792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573001.046504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573001.046508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573001.046510:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495473664:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573001.046525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573001.046532:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1217 00000100:00080000:2.0:1788573001.046537:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573001.046542:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573001.046548:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573001.046564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495473920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573001.046577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573001.046581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573001.046583:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573001.046587:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1217 00000100:00080000:2.0:1788573001.046592:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573001.046611:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573001.046618:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573001.046625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573001.046640:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495474048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573001.046675:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff93a24940b800 x1875453495473664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573001.046699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495473664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573001.046711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573001.046716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573001.046720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495473664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573001.046758:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573001.046785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573001.046800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff93a249409c00 x1875453495473792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573001.046814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495473792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573001.046830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573001.046834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573001.046838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573001.046844:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473920:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573001.046870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573001.046883:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff93a24940a680 x1875453495473920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573001.046895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495473920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573001.046904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573001.046907:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573001.046911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495473920:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573001.046917:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495474048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573001.046938:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573001.046950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff93a249408e00 x1875453495474048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573001.046971:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495474048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573001.046996:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573001.047001:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573001.047006:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495474048:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573002.006372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573003.030387:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573004.054359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573005.078404:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573006.102414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573006.166348:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573006.166358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573006.166362:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573006.166370:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1222 00000100:00080000:0.0:1788573006.166379:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573006.166385:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573006.166418:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495474176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573006.166440:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573006.166443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573006.166445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573006.166448:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1222 00000100:00080000:0.0:1788573006.166450:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573006.166453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573006.166460:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573006.166470:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495474304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573006.166479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573006.166481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573006.166482:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573006.166498:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1222 00000100:00080000:0.0:1788573006.166502:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573006.166511:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573006.166513:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495474176:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573006.166518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573006.166540:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495474432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573006.166556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573006.166560:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573006.166562:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573006.166568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1222 00000100:00080000:0.0:1788573006.166571:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573006.166575:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573006.166579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573006.166590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495474560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573006.166626:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573006.166696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff93a2460d0700 x1875453495474176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573006.166721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495474176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573006.166737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573006.166742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573006.166748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495474176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573006.166757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573006.166796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573006.166812:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff93a2460d1c00 x1875453495474304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573006.166828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495474304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573006.166860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573006.166864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573006.166869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573006.166901:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474432:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573006.166929:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573006.166942:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff93a2460d0000 x1875453495474432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573006.166956:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495474432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573006.166966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573006.166970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573006.166987:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474432:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573006.166996:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495474560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573006.167020:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573006.167048:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff93a2460d0a80 x1875453495474560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573006.167062:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495474560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573006.167085:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573006.167089:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573006.167100:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495474560:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573007.126412:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573008.150385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573009.174442:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573010.198429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573011.222418:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573011.286294:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573011.286303:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573011.286307:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573011.286315:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1227 00000100:00080000:0.0:1788573011.286324:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573011.286329:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573011.286360:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495474688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573011.286382:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573011.286396:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573011.286398:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573011.286406:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1227 00000100:00080000:0.0:1788573011.286410:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573011.286414:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573011.286422:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573011.286433:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495474816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573011.286440:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495474688:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573011.286443:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573011.286450:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573011.286452:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573011.286456:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1227 00000100:00080000:0.0:1788573011.286459:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573011.286462:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573011.286467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573011.286481:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495474944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573011.286492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573011.286496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573011.286499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573011.286503:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1227 00000100:00080000:0.0:1788573011.286508:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573011.286512:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573011.286523:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573011.286533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573011.286543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495475072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573011.286570:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff93a2460d3b80 x1875453495474688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573011.286592:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495474688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573011.286604:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573011.286607:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573011.286610:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495474688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573011.286630:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573011.286649:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573011.286661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff93a2460d0380 x1875453495474816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573011.286674:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495474816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573011.286682:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573011.286686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573011.286690:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573011.286696:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474944:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573011.286718:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573011.286734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff93a2460d3480 x1875453495474944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573011.286743:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495474944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573011.286748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573011.286751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573011.286753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495474944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573011.286757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495475072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573011.286770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573011.286776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff93a2460d1500 x1875453495475072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573011.286784:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495475072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573011.286789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573011.286790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573011.286792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495475072:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573012.246467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573013.270491:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573014.294414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573015.318393:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573016.342379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573016.406360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.406372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573016.406378:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.406389:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1232 00000100:00080000:3.0:1788573016.406409:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.406417:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573016.406468:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495475200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.406497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.406501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573016.406503:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.406507:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1232 00000100:00080000:3.0:1788573016.406512:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.406517:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573016.406528:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573016.406543:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495475200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573016.406546:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495475328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.406560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.406564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573016.406566:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.406570:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1232 00000100:00080000:3.0:1788573016.406575:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.406580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573016.406586:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573016.406608:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573016.406612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495475456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.406626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.406630:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573016.406633:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.406644:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1232 00000100:00080000:3.0:1788573016.406650:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573016.406653:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff93a25064e300 x1875453495475200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573016.406659:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573016.406670:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573016.406677:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495475200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573016.406687:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495475584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573016.406693:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573016.406703:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573016.406710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495475200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573016.406738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573016.406758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573016.406767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff93a25064fb80 x1875453495475328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573016.406776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495475328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573016.406782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573016.406785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573016.406788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573016.406805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475456:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573016.406850:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573016.406866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff93a25064d180 x1875453495475456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573016.406881:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495475456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573016.406889:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573016.406893:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573016.406897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475456:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573016.406903:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495475584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573016.406937:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573016.406949:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff93a25064c000 x1875453495475584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573016.406962:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495475584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573016.406971:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573016.406974:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573016.406978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495475584:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573017.366424:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573018.390388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573019.414448:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573020.438392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573021.462379:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573021.526352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.526364:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573021.526369:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.526379:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1237 00000100:00080000:3.0:1788573021.526389:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.526397:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573021.526434:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495475712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.526466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.526470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573021.526472:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.526477:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1237 00000100:00080000:3.0:1788573021.526481:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.526487:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573021.526497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573021.526513:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495475712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573021.526514:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495475840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.526532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.526550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573021.526552:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.526558:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1237 00000100:00080000:3.0:1788573021.526573:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573021.526580:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573021.526581:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573021.526591:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573021.526607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495475968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.526619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.526623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1788573021.526628:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff93a25064d880 x1875453495475712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573021.526628:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.526639:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1237 00000100:00080000:3.0:1788573021.526646:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.526651:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573021.526657:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495475712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573021.526658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788573021.526675:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573021.526677:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495476096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573021.526685:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573021.526694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495475712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573021.526725:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573021.526765:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573021.526776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff93a25064e680 x1875453495475840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573021.526785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495475840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573021.526790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573021.526793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573021.526795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573021.526799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475968:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573021.526814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573021.526820:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff93a25064ed80 x1875453495475968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573021.526834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495475968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573021.526840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573021.526842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573021.526844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495475968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573021.526847:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495476096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573021.526859:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573021.526865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff93a25064dc00 x1875453495476096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573021.526873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495476096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573021.526878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573021.526881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573021.526884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495476096:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573022.486410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573023.510484:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573024.534385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573025.558388:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573026.582405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573026.646347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.646355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573026.646359:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.646389:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1243 00000100:00080000:2.0:1788573026.646398:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.646404:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573026.646430:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495476224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573026.646451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.646454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573026.646456:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.646459:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1243 00000100:00080000:2.0:1788573026.646461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.646464:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573026.646472:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573026.646480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495476352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573026.646491:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495476224:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573026.646494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.646501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573026.646503:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.646508:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1243 00000100:00080000:2.0:1788573026.646512:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.646516:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573026.646522:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573026.646539:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495476480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573026.646558:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573026.646560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.646566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573026.646568:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.646574:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1243 00000100:00080000:2.0:1788573026.646580:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.646584:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573026.646590:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573026.646598:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff93a24940ad80 x1875453495476224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573026.646619:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495476224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573026.646620:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495476608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573026.646644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573026.646671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573026.646676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495476224:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573026.646704:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573026.646729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573026.646743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff93a249408000 x1875453495476352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573026.646756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495476352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573026.646762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573026.646764:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573026.646767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573026.646771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476480:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573026.646786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573026.646793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff93a249409180 x1875453495476480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573026.646801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495476480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573026.646810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573026.646813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573026.646816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476480:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573026.646830:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495476608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573026.646851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573026.646862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff93a249408a80 x1875453495476608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573026.646870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495476608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573026.646875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573026.646877:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573026.646880:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495476608:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573027.606352:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573028.630411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573029.654376:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573030.678398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573031.702383:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573031.766336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.766348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573031.766352:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.766363:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1248 00000100:00080000:2.0:1788573031.766382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.766391:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573031.766432:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495476736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573031.766461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.766465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573031.766482:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.766488:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1248 00000100:00080000:2.0:1788573031.766494:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.766499:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573031.766509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573031.766524:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495476736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573031.766528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495476864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573031.766544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.766548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573031.766550:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.766570:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1248 00000100:00080000:2.0:1788573031.766575:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.766580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573031.766585:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573031.766599:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495476992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573031.766604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573031.766616:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.766620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573031.766623:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.766628:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1248 00000100:00080000:2.0:1788573031.766632:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.766636:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573031.766641:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573031.766660:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff93a24940aa00 x1875453495476736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573031.766666:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495477120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573031.766687:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495476736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573031.766697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573031.766701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573031.766705:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495476736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573031.766724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573031.766741:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573031.766750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff93a249408700 x1875453495476864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573031.766767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495476864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573031.766772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573031.766775:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573031.766777:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573031.766781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476992:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573031.766795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573031.766802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff93a249409880 x1875453495476992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573031.766810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495476992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573031.766815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573031.766816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573031.766818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495476992:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573031.766822:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495477120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573031.766832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573031.766837:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff93a249409500 x1875453495477120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573031.766844:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495477120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573031.766859:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573031.766861:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573031.766863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495477120:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573032.726412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573033.750389:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573034.774460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573035.798431:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573036.822459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573036.886406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573036.886417:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573036.886422:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573036.886434:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1253 00000100:00080000:3.0:1788573036.886445:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573036.886453:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573036.886516:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495477248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573036.886545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573036.886548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573036.886550:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573036.886553:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1253 00000100:00080000:3.0:1788573036.886556:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573036.886559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573036.886570:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573036.886581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495477376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573036.886591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573036.886599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788573036.886605:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495477248:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573036.886622:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573036.886629:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1253 00000100:00080000:3.0:1788573036.886632:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573036.886637:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573036.886642:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573036.886659:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495477504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573036.886667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573036.886671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573036.886673:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573036.886676:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1253 00000100:00080000:3.0:1788573036.886680:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573036.886687:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:2.0:1788573036.886688:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788573036.886700:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573036.886711:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495477632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573036.886742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff93a25064ea00 x1875453495477248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573036.886759:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495477248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573036.886768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573036.886771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573036.886774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495477248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573036.886797:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495477376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573036.886813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573036.886830:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff93a25064dc00 x1875453495477376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573036.886839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495477376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573036.886844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573036.886860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573036.886863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495477376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573036.886868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495477504:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573036.886884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573036.886891:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff93a25064ed80 x1875453495477504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573036.886899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495477504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573036.886904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573036.886906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573036.886907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495477504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573036.886911:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495477632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573036.886922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573036.886928:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff93a25064e680 x1875453495477632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573036.886935:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495477632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573036.886940:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573036.886941:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573036.886943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495477632:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573037.846417:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573038.870395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573039.894379:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573040.918417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573041.942403:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573042.006348:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573042.006363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573042.006380:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573042.006392:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1258 00000100:00080000:0.0:1788573042.006409:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573042.006420:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573042.006472:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495477760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573042.006508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573042.006512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573042.006514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573042.006519:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1258 00000100:00080000:0.0:1788573042.006523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573042.006528:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573042.006540:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573042.006557:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495477888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573042.006570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573042.006575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573042.006578:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573042.006584:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495477760:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573042.006588:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1258 00000100:00080000:0.0:1788573042.006616:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573042.006625:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573042.006633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573042.006649:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495478016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573042.006656:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573042.006683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573042.006688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573042.006690:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573042.006700:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1258 00000100:00100000:2.0:1788573042.006700:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff93a2460d2680 x1875453495477760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573042.006713:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573042.006724:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573042.006731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573042.006736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495477760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573042.006749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573042.006754:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495478144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573042.006757:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573042.006767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495477760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573042.006805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495477888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573042.006841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573042.006856:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff93a2460d2a00 x1875453495477888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573042.006875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495477888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573042.006883:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573042.006886:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573042.006888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495477888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573042.006894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478016:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573042.006924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573042.006937:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff93a2460d1180 x1875453495478016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573042.006949:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495478016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573042.006958:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573042.006962:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573042.006966:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478016:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573042.006983:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495478144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573042.007006:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573042.007018:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff93a2460d2300 x1875453495478144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573042.007027:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495478144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573042.007033:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573042.007035:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573042.007038:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495478144:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573042.966412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573043.990431:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573045.014380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573046.038430:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573047.062399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573047.126352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573047.126363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573047.126368:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573047.126379:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1263 00000100:00080000:0.0:1788573047.126390:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573047.126399:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573047.126434:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495478272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573047.126458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573047.126475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573047.126477:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573047.126480:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1263 00000100:00080000:0.0:1788573047.126484:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573047.126487:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573047.126495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573047.126506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495478400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573047.126515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573047.126518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495478272:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573047.126519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573047.126520:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573047.126524:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1263 00000100:00080000:0.0:1788573047.126527:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573047.126532:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573047.126537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573047.126546:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495478528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573047.126553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573047.126556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573047.126557:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573047.126560:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1263 00000100:00080000:0.0:1788573047.126562:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573047.126564:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573047.126574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573047.126581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495478656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573047.126598:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573047.126642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff93a2460d1f80 x1875453495478272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573047.126664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495478272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573047.126675:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573047.126679:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573047.126683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495478272:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573047.126707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573047.126731:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573047.126744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff93a2460d2300 x1875453495478400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573047.126757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495478400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573047.126765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573047.126769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573047.126772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573047.126779:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478528:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573047.126799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573047.126811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff93a2460d1180 x1875453495478528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573047.126838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495478528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573047.126846:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573047.126849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573047.126852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573047.126859:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495478656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573047.126880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573047.126892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff93a2460d2a00 x1875453495478656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573047.126913:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495478656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573047.126922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573047.126926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573047.126929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495478656:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573048.086436:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573049.110430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573050.134401:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573051.158373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573052.182420:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573052.246316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573052.246325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573052.246329:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573052.246337:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1268 00000100:00080000:0.0:1788573052.246345:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573052.246352:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573052.246394:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495478784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573052.246425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573052.246429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573052.246432:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573052.246436:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1268 00000100:00080000:0.0:1788573052.246441:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573052.246446:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573052.246457:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573052.246469:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495478784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573052.246475:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495478912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573052.246489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573052.246493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573052.246495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573052.246500:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1268 00000100:00080000:0.0:1788573052.246528:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573052.246534:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573052.246542:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573052.246554:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573052.246571:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495479040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573052.246578:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff93a2460d2680 x1875453495478784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573052.246585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573052.246593:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573052.246598:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573052.246606:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1268 00000100:00100000:2.0:1788573052.246611:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495478784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573052.246612:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573052.246621:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:2.0:1788573052.246625:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788573052.246627:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788573052.246631:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573052.246635:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495478784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573052.246641:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495479168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573052.246658:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573052.246681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573052.246692:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff93a2460d1500 x1875453495478912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573052.246701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495478912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573052.246710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573052.246713:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573052.246717:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495478912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573052.246723:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479040:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573052.246746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573052.246765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff93a2460d3480 x1875453495479040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573052.246775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495479040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573052.246790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573052.246793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573052.246795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479040:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573052.246799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495479168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573052.246814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573052.246821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff93a2460d0380 x1875453495479168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573052.246829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495479168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573052.246834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573052.246835:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573052.246837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495479168:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573053.206406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573054.230382:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573055.254333:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573056.278423:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573057.302361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573057.366324:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.366333:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573057.366336:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.366345:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1273 00000100:00080000:2.0:1788573057.366355:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.366363:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573057.366419:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495479296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.366450:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.366453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573057.366454:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.366457:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1273 00000100:00080000:2.0:1788573057.366461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.366464:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573057.366472:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573057.366483:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495479424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.366492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573057.366494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495479296:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573057.366495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573057.366499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.366505:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1273 00000100:00080000:2.0:1788573057.366507:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.366509:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573057.366512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573057.366521:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495479552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.366529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.366532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573057.366533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.366542:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1273 00000100:00080000:2.0:1788573057.366546:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.366549:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573057.366553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573057.366557:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573057.366566:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495479680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573057.366609:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff93a24940b480 x1875453495479296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573057.366625:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495479296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573057.366633:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573057.366643:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573057.366646:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495479296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573057.366664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573057.366682:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573057.366691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff93a24940bb80 x1875453495479424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573057.366699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495479424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573057.366704:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573057.366706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573057.366709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573057.366712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479552:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573057.366723:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573057.366728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff93a24940b100 x1875453495479552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573057.366735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495479552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573057.366740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573057.366742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573057.366744:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479552:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573057.366747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495479680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573057.366764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573057.366770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff93a249409500 x1875453495479680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573057.366778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495479680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573057.366783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573057.366784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573057.366786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495479680:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573058.326396:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573059.350441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573060.374413:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573061.398363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573062.423122:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573062.486328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.486339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573062.486344:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.486355:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1278 00000100:00080000:0.0:1788573062.486366:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.486373:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573062.486412:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495479808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.486444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.486465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573062.486468:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.486474:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1278 00000100:00080000:0.0:1788573062.486480:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.486485:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573062.486493:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495479808:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573062.486496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573062.486518:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495479936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.486530:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.486534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573062.486536:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.486541:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1278 00000100:00080000:0.0:1788573062.486546:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.486551:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573062.486557:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573062.486570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495480064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.486582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.486586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573062.486588:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.486593:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1278 00000100:00080000:0.0:1788573062.486598:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573062.486603:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479936:192.168.202.102@tcp:38:kworker.0 00000100:00080000:0.0:1788573062.486605:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573062.486620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573062.486638:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573062.486640:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495480192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573062.486666:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495480192:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573063.446952:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573064.471619:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573065.495188:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573065.558529:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788573065.558576:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573065.558594:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573065.558637:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573065.558648:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573065.558656:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573065.558686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff93a2460d1c00 x1875453495480192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573065.558711:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495480192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573065.558720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573065.558724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573065.558729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495480192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573065.558738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff93a2460d0a80 x1875453495479936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573065.558771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495479936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573065.558780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573065.558790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573065.558793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495479936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573065.558801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff93a2460d0000 x1875453495480064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573065.558813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495480064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573065.558820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573065.558824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573065.558827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573065.558834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff93a2460d3b80 x1875453495479808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573065.558845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495479808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573065.558853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573065.558856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573065.558859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495479808:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788573066.518696:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573067.542480:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573067.606380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573067.606395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573067.606401:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573067.606415:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1284 00000100:00080000:2.0:1788573067.606430:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573067.606440:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573067.606486:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495480320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573067.606518:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573067.606521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573067.606541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573067.606547:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1284 00000100:00080000:2.0:1788573067.606553:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573067.606559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573067.606572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573067.606580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495480320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573067.606590:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495480448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573067.606608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573067.606612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573067.606614:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573067.606619:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1284 00000100:00080000:2.0:1788573067.606623:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573067.606628:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573067.606635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573067.606649:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495480576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573067.606660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573067.606664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573067.606665:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573067.606670:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1284 00000100:00080000:2.0:1788573067.606675:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573067.606679:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573067.606684:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573067.606694:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573067.606709:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495480704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573067.606775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff93a249409880 x1875453495480320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573067.606805:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495480320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573067.606821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573067.606827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573067.606833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495480320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573067.606879:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573067.606948:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573067.606967:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff93a249408700 x1875453495480448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573067.606983:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495480448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573067.606993:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573067.606999:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573067.607003:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573067.607011:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480576:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573067.607038:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573067.607052:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff93a24940aa00 x1875453495480576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573067.607084:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495480576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573067.607095:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573067.607099:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573067.607103:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480576:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573067.607110:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495480704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573067.607135:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573067.607149:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff93a249408a80 x1875453495480704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573067.607163:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495480704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573067.607171:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573067.607176:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573067.607180:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495480704:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573068.566444:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573069.590526:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573070.614450:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573071.638462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573072.662417:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573072.726353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573072.726361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573072.726366:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573072.726375:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1289 00000100:00080000:2.0:1788573072.726385:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573072.726392:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573072.726428:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495480832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573072.726449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573072.726452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573072.726453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573072.726456:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1289 00000100:00080000:2.0:1788573072.726458:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573072.726461:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573072.726469:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573072.726480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495480960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573072.726487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573072.726503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573072.726509:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573072.726511:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495480832:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573072.726520:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1289 00000100:00080000:2.0:1788573072.726531:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573072.726540:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573072.726546:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573072.726563:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495481088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573072.726577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573072.726582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573072.726584:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573072.726589:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1289 00000100:00080000:2.0:1788573072.726597:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573072.726601:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573072.726606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573072.726623:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573072.726624:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495481216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573072.726684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff93a249409180 x1875453495480832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573072.726703:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495480832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573072.726727:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573072.726731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573072.726735:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495480832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573072.726761:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573072.726784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573072.726795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff93a249408000 x1875453495480960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573072.726804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495480960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573072.726809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573072.726812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573072.726814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495480960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573072.726838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481088:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573072.726852:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573072.726858:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff93a24940ad80 x1875453495481088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573072.726866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495481088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573072.726871:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573072.726873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573072.726875:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481088:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573072.726890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495481216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573072.726905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573072.726912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff93a249408e00 x1875453495481216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573072.726920:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495481216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573072.726925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573072.726927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573072.726929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495481216:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573073.686424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573074.710443:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573075.734401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573076.758405:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573077.782457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573077.846356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573077.846366:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573077.846370:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573077.846378:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1294 00000100:00080000:0.0:1788573077.846389:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573077.846397:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573077.846437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495481344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573077.846479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573077.846483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573077.846485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573077.846490:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1294 00000100:00080000:0.0:1788573077.846493:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573077.846496:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573077.846505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573077.846533:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495481344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573077.846534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495481472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573077.846560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573077.846566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573077.846568:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573077.846572:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1294 00000100:00080000:0.0:1788573077.846591:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573077.846596:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573077.846602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573077.846617:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495481600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573077.846631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573077.846634:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788573077.846637:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573077.846638:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573077.846648:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1294 00000100:00080000:0.0:1788573077.846652:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573077.846655:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573077.846660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573077.846670:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495481728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573077.846691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573077] req@ffff93a2460d0700 x1875453495481344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573077.846712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495481344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573077.846724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573077.846728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573077.846731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495481344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573077.846754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573077.846778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573077.846791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573077] req@ffff93a2460d3b80 x1875453495481472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573077.846813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495481472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573077.846821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573077.846824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573077.846827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573077.846833:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481600:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573077.846853:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573077.846862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573077] req@ffff93a2460d0000 x1875453495481600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573077.846874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495481600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573077.846882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573077.846885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573077.846889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573077.846894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495481728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573077.846910:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573077.846920:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573077] req@ffff93a2460d0a80 x1875453495481728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573077.846931:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495481728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573077.846951:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573077.846955:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573077.846959:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495481728:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573078.806359:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573079.830379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573080.854331:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573081.878440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573082.902469:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573082.966301:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.966310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573082.966313:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.966322:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1299 00000100:00080000:0.0:1788573082.966330:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573082.966336:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573082.966390:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495481856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573082.966413:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.966431:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573082.966434:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.966441:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1299 00000100:00080000:0.0:1788573082.966446:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573082.966451:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573082.966458:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495481856:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573082.966462:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573082.966481:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495481984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573082.966493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.966496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573082.966497:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.966500:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1299 00000100:00080000:0.0:1788573082.966511:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573082.966516:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573082.966519:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573082.966525:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573082.966536:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495482112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573082.966544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.966547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573082.966548:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.966551:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1299 00000100:00080000:0.0:1788573082.966555:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573082.966558:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff93a2460d1c00 x1875453495481856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573082.966560:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573082.966572:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573082.966577:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495481856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573082.966584:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495482240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573082.966589:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573082.966598:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573082.966603:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495481856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573082.966623:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573082.966640:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573082.966648:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff93a2460d0380 x1875453495481984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573082.966656:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495481984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573082.966662:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573082.966664:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573082.966666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495481984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573082.966677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495482112:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573082.966692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573082.966707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff93a2460d3480 x1875453495482112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573082.966718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495482112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573082.966724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573082.966726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573082.966728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495482112:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573082.966732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495482240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573082.966744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573082.966751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff93a2460d1500 x1875453495482240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573082.966758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495482240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573082.966763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573082.966765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573082.966767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495482240:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573083.926398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573084.950404:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573085.974366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573086.998389:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573088.022357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573088.086382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573088.086393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573088.086398:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573088.086409:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1304 00000100:00080000:2.0:1788573088.086419:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573088.086427:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573088.086465:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495482368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573088.086495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573088.086499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573088.086501:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573088.086506:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1304 00000100:00080000:2.0:1788573088.086509:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573088.086514:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573088.086524:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573088.086542:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495482368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573088.086543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495482496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573088.086559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573088.086563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573088.086579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573088.086584:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1304 00000100:00080000:2.0:1788573088.086590:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573088.086599:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573088.086607:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573088.086619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573088.086634:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495482624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573088.086639:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff93a24940a680 x1875453495482368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573088.086650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573088.086658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573088.086660:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573088.086665:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495482368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573088.086667:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1304 00000100:00080000:2.0:1788573088.086676:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573088.086679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573088.086684:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:3.0:1788573088.086686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573088.086693:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495482368:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573088.086695:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573088.086713:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495482752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573088.086716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495482496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573088.086743:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573088.086753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff93a249409c00 x1875453495482496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573088.086762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495482496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573088.086768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573088.086770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573088.086772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495482496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573088.086776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495482624:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573088.086787:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573088.086793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff93a24940b800 x1875453495482624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573088.086800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495482624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573088.086814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573088.086817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573088.086819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495482624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573088.086829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495482752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573088.086842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573088.086849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff93a249408e00 x1875453495482752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573088.086856:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495482752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573088.086862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573088.086863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573088.086865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495482752:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573089.046433:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573090.070363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573091.094377:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573092.118401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573093.142383:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573093.206352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573093.206367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573093.206371:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573093.206380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1309 00000100:00080000:0.0:1788573093.206389:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573093.206396:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573093.206428:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495482880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573093.206449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573093.206451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573093.206453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573093.206456:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1309 00000100:00080000:0.0:1788573093.206458:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573093.206460:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573093.206468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573093.206476:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495483008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573093.206482:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573093.206484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573093.206485:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573093.206489:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1309 00000100:00080000:0.0:1788573093.206504:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573093.206508:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573093.206512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573093.206514:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495482880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573093.206524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495483136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573093.206540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573093.206544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573093.206550:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573093.206555:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1309 00000100:00080000:0.0:1788573093.206567:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573093.206572:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573093.206576:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573093.206584:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573093.206600:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495483264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573093.206617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff93a2460d2680 x1875453495482880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573093.206640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495482880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573093.206652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573093.206655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573093.206658:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495482880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573093.206673:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573093.206692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573093.206702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff93a2460d2a00 x1875453495483008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573093.206710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495483008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573093.206715:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573093.206718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573093.206720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573093.206737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483136:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573093.206748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573093.206754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff93a2460d1180 x1875453495483136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573093.206761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495483136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573093.206767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573093.206768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573093.206770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483136:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573093.206774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495483264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573093.206783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573093.206789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff93a2460d2300 x1875453495483264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573093.206801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495483264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573093.206806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573093.206808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573093.206810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495483264:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573094.166387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573095.190383:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573096.214394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573097.238419:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573098.262421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573098.326301:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573098.326312:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573098.326316:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573098.326329:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1314 00000100:00080000:0.0:1788573098.326340:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573098.326347:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573098.326387:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495483392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573098.326414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573098.326438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573098.326439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573098.326443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1314 00000100:00080000:0.0:1788573098.326446:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573098.326450:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573098.326461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573098.326474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495483520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573098.326482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573098.326485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573098.326486:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573098.326490:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1314 00000100:00080000:0.0:1788573098.326492:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573098.326495:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573098.326499:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573098.326508:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495483392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573098.326510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495483648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573098.326534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573098.326537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573098.326538:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573098.326542:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1314 00000100:00080000:0.0:1788573098.326545:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573098.326548:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573098.326558:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573098.326570:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495483776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573098.326590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573098.326630:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff93a2460d1f80 x1875453495483392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573098.326644:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495483392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573098.326653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573098.326657:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573098.326660:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495483392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573098.326666:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573098.326680:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573098.326688:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff93a2460d2300 x1875453495483520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573098.326696:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495483520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573098.326721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573098.326725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573098.326729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573098.326788:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483648:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573098.326808:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573098.326817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff93a2460d1180 x1875453495483648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573098.326825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495483648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573098.326837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573098.326839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573098.326842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495483648:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573098.326846:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495483776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573098.326858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573098.326865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff93a2460d2a00 x1875453495483776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573098.326873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495483776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573098.326878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573098.326880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573098.326888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495483776:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573099.286447:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573100.310351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573101.334394:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573102.358410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573103.382371:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573103.446364:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.446373:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573103.446377:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.446385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1319 00000100:00080000:2.0:1788573103.446394:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.446400:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573103.446431:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495483904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573103.446456:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.446459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573103.446460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.446463:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1319 00000100:00080000:2.0:1788573103.446466:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.446468:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573103.446477:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573103.446485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495484032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573103.446493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.446495:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573103.446497:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.446512:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1319 00000100:00080000:2.0:1788573103.446520:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573103.446522:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495483904:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573103.446530:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573103.446539:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573103.446558:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495484160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573103.446587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.446591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573103.446595:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.446599:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1319 00000100:00080000:2.0:1788573103.446612:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.446617:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573103.446623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573103.446627:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573103.446645:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495484288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573103.446683:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff93a24940ad80 x1875453495483904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573103.446704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495483904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573103.446726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573103.446732:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573103.446737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495483904:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573103.446746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573103.446772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573103.446782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff93a249408000 x1875453495484032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573103.446791:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495484032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573103.446797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573103.446799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573103.446802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573103.446827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484160:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573103.446841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573103.446863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff93a249409180 x1875453495484160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573103.446872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495484160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573103.446878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573103.446880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573103.446882:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573103.446886:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495484288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573103.446898:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573103.446905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff93a249408a80 x1875453495484288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573103.446913:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495484288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573103.446918:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573103.446919:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573103.446921:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495484288:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573104.406423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573105.430344:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573106.454411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573107.478445:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573108.502457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573108.566320:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573108.566330:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573108.566334:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573108.566343:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1325 00000100:00080000:2.0:1788573108.566351:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573108.566389:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573108.566420:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495484416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573108.566446:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573108.566449:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573108.566450:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573108.566453:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1325 00000100:00080000:2.0:1788573108.566457:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573108.566459:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573108.566467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573108.566476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495484544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573108.566484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573108.566486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573108.566488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573108.566490:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1325 00000100:00080000:2.0:1788573108.566492:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573108.566498:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573108.566499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495484416:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573108.566502:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573108.566512:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495484672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573108.566520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573108.566524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573108.566532:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573108.566537:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1325 00000100:00080000:2.0:1788573108.566551:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573108.566556:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788573108.566561:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:2.0:1788573108.566565:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573108.566582:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495484800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573108.566602:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff93a24940aa00 x1875453495484416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573108.566618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495484416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573108.566633:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573108.566636:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573108.566639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495484416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573108.566645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573108.566663:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573108.566672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff93a249408700 x1875453495484544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573108.566680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495484544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573108.566686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573108.566688:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573108.566690:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573108.566708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484672:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573108.566719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573108.566725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff93a249409880 x1875453495484672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573108.566733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495484672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573108.566738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573108.566740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573108.566742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495484672:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573108.566751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495484800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573108.566763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573108.566770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff93a249409500 x1875453495484800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573108.566778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495484800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573108.566783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573108.566785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573108.566787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495484800:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573109.526417:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573110.550422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573111.574438:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573112.598403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573113.622381:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573113.686316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.686326:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573113.686330:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.686339:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1330 00000100:00080000:1.0:1788573113.686347:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.686353:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573113.686385:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495484928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.686426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.686429:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573113.686431:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.686434:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1330 00000100:00080000:1.0:1788573113.686437:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.686441:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573113.686449:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573113.686458:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495485056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.686466:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.686469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573113.686470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.686473:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1330 00000100:00080000:1.0:1788573113.686475:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.686478:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573113.686481:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573113.686485:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495484928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573113.686491:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495485184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.686500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.686503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573113.686504:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.686506:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1330 00000100:00080000:1.0:1788573113.686516:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.686519:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573113.686524:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573113.686540:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495485312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573113.686549:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573113.686600:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff93a258da0700 x1875453495484928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573113.686617:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495484928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573113.686626:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573113.686629:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573113.686632:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495484928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573113.686638:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573113.686655:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573113.686664:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff93a258da1c00 x1875453495485056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573113.686679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495485056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573113.686685:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573113.686687:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573113.686689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573113.686708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485184:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573113.686722:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573113.686729:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff93a258da2680 x1875453495485184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573113.686737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495485184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573113.686741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573113.686743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573113.686745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573113.686748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495485312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573113.686758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573113.686764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff93a258da1880 x1875453495485312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573113.686771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495485312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573113.686785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573113.686787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573113.686789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495485312:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573114.646469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573115.670424:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573116.694388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573117.718360:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573118.742399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573118.806349:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573118.806359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573118.806362:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573118.806370:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1335 00000100:00080000:3.0:1788573118.806379:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573118.806385:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573118.806416:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495485440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573118.806439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573118.806443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573118.806445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573118.806449:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1335 00000100:00080000:3.0:1788573118.806453:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573118.806458:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573118.806467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573118.806480:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495485568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573118.806484:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495485440:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573118.806490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573118.806494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573118.806511:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573118.806516:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1335 00000100:00080000:3.0:1788573118.806520:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573118.806524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573118.806529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573118.806540:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495485696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573118.806549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573118.806553:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573118.806553:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573118.806555:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573118.806558:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1335 00000100:00080000:3.0:1788573118.806560:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573118.806563:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573118.806567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573118.806576:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495485824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573118.806601:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff93a25064d880 x1875453495485440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573118.806619:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495485440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573118.806631:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573118.806635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573118.806638:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495485440:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573118.806664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573118.806684:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573118.806693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff93a25064c000 x1875453495485568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573118.806702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495485568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573118.806708:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573118.806710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573118.806713:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573118.806716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485696:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573118.806727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573118.806733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff93a25064d180 x1875453495485696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573118.806740:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495485696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573118.806746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573118.806748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573118.806749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495485696:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573118.806752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495485824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573118.806762:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573118.806767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff93a25064fb80 x1875453495485824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573118.806780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495485824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573118.806786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573118.806788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573118.806790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495485824:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573119.766335:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573120.790386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573121.814422:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573122.838363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573123.862430:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573123.926417:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573123.926427:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573123.926433:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573123.926444:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1340 00000100:00080000:2.0:1788573123.926455:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573123.926463:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573123.926502:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495485952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573123.926537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573123.926542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573123.926545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573123.926550:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1340 00000100:00080000:2.0:1788573123.926554:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573123.926576:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573123.926587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573123.926604:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495485952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573123.926605:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495486080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573123.926635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573123.926639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573123.926641:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573123.926648:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1340 00000100:00080000:2.0:1788573123.926653:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573123.926658:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573123.926665:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573123.926680:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495486208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573123.926693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573123.926697:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573123.926699:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573123.926703:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1340 00000100:00080000:2.0:1788573123.926707:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573123.926714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573123.926721:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573123.926728:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573123.926746:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495486336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573123.926810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff93a24940b100 x1875453495485952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573123.926837:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495485952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573123.926850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573123.926868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573123.926873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495485952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573123.926904:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573123.926931:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573123.926944:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff93a24940bb80 x1875453495486080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573123.926958:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495486080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573123.926967:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573123.926971:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573123.926975:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573123.926982:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486208:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573123.927002:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573123.927013:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff93a24940b480 x1875453495486208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573123.927025:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495486208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573123.927034:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573123.927037:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573123.927041:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486208:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573123.927047:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495486336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573123.927075:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573123.927087:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff93a249409500 x1875453495486336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573123.927100:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495486336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573123.927108:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573123.927112:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573123.927115:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495486336:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573124.886417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573125.910477:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573126.934475:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573127.958436:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573128.984935:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573129.046367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573129.046379:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573129.046384:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573129.046395:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1345 00000100:00080000:0.0:1788573129.046406:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573129.046413:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573129.046453:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495486464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573129.046480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573129.046484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573129.046486:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573129.046490:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1345 00000100:00080000:0.0:1788573129.046494:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573129.046498:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573129.046516:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573129.046529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495486592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573129.046540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573129.046543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573129.046546:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573129.046553:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1345 00000100:00080000:0.0:1788573129.046574:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573129.046579:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573129.046601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573129.046617:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495486464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573129.046620:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495486720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573129.046649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573129.046654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573129.046656:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573129.046662:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1345 00000100:00080000:0.0:1788573129.046667:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573129.046673:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573129.046681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573129.046697:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495486848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573129.046762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573129.046803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573129.046821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495486848:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573130.006774:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573131.030567:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573132.054828:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788573132.118744:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788573132.118810:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573132.118836:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573132.118901:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573132.118913:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573132.118930:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573132.119207:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff93a2460d3480 x1875453495486720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573132.119279:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495486720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573132.119297:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573132.119305:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573132.119311:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573132.119320:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff93a2460d0380 x1875453495486848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573132.119372:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495486848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573132.119381:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573132.119386:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573132.119389:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495486848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573132.119397:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff93a2460d2680 x1875453495486464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573132.120021:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495486464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573132.120032:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573132.120037:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573132.120040:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495486464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573132.120049:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff93a2460d1500 x1875453495486592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573132.120065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495486592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573132.120073:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573132.120077:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573132.120080:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495486592:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573133.078675:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573134.102444:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573134.166449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.166465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573134.166471:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.166487:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1350 00000100:00080000:2.0:1788573134.166500:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.166509:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573134.166558:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495486976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573134.166590:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.166594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573134.166596:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.166602:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1350 00000100:00080000:2.0:1788573134.166607:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.166612:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573134.166625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573134.166639:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495487104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573134.166646:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495486976:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573134.166650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.166654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573134.166668:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.166677:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1350 00000100:00080000:2.0:1788573134.166682:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.166687:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573134.166694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573134.166709:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495487232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573134.166722:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.166726:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573134.166727:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.166732:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1350 00000100:00080000:2.0:1788573134.166759:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573134.166766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573134.166767:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573134.166777:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573134.166791:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495487360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573134.166852:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff93a249409880 x1875453495486976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573134.166880:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495486976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573134.166894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573134.166899:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573134.166905:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495486976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573134.166945:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573134.166972:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573134.166989:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff93a249408700 x1875453495487104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573134.167003:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495487104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573134.167012:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573134.167015:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573134.167019:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573134.167031:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487232:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573134.167054:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573134.167067:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff93a24940aa00 x1875453495487232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573134.167113:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495487232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573134.167122:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573134.167125:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573134.167128:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487232:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573134.167134:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495487360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573134.167292:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573134.167308:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff93a249408a80 x1875453495487360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573134.167322:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495487360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573134.167331:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573134.167335:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573134.167338:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495487360:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573135.126442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573136.150442:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573137.174370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573138.198426:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573139.222375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573139.286312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573139.286332:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573139.286335:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573139.286354:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1355 00000100:00080000:0.0:1788573139.286363:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573139.286370:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573139.286400:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495487488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573139.286420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573139.286423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573139.286424:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573139.286427:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1355 00000100:00080000:0.0:1788573139.286430:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573139.286433:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573139.286441:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573139.286452:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495487616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573139.286454:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495487488:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573139.286468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573139.286471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573139.286472:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573139.286475:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1355 00000100:00080000:0.0:1788573139.286477:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573139.286480:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573139.286483:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573139.286491:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495487744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573139.286505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573139.286507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573139.286508:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573139.286511:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1355 00000100:00080000:0.0:1788573139.286514:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573139.286517:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573139.286520:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573139.286528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495487872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573139.286540:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573139.286582:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff93a2460d1c00 x1875453495487488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573139.286597:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495487488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573139.286615:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573139.286618:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573139.286621:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495487488:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573139.286640:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573139.286660:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573139.286669:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff93a2460d0a80 x1875453495487616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573139.286678:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495487616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573139.286683:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573139.286686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573139.286688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573139.286691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487744:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573139.286703:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573139.286709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff93a2460d0000 x1875453495487744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573139.286716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495487744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573139.286721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573139.286723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573139.286725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495487744:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573139.286734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495487872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573139.286756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573139.286765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff93a2460d3b80 x1875453495487872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573139.286777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495487872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573139.286786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573139.286789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573139.286793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495487872:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573140.246412:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573141.270466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573142.294427:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573143.318451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573144.342461:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573144.406362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573144.406377:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573144.406383:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573144.406397:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1360 00000100:00080000:0.0:1788573144.406413:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573144.406422:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573144.406467:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495488000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573144.406496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573144.406500:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573144.406501:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573144.406530:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1360 00000100:00080000:0.0:1788573144.406533:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573144.406537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573144.406548:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573144.406561:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495488128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573144.406570:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495488000:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573144.406573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573144.406577:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573144.406578:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573144.406583:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1360 00000100:00080000:0.0:1788573144.406587:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573144.406590:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573144.406593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573144.406601:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495488256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573144.406610:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573144.406613:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573144.406614:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573144.406617:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1360 00000100:00080000:0.0:1788573144.406619:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573144.406622:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573144.406624:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573144.406631:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495488384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573144.406672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573144.406736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff93a2460d0700 x1875453495488000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573144.406754:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495488000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573144.406764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573144.406768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573144.406771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495488000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573144.406795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573144.406814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573144.406823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff93a2460d3b80 x1875453495488128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573144.406840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495488128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573144.406847:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573144.406850:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573144.406854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573144.406861:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488256:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573144.406886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573144.406899:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff93a2460d0000 x1875453495488256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573144.406909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495488256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573144.406915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573144.406917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573144.406920:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573144.406925:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495488384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573144.406942:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573144.406951:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff93a2460d0a80 x1875453495488384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573144.406959:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495488384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573144.406974:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573144.406977:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573144.406979:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495488384:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573145.366460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573146.390392:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573147.414386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573148.438367:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573149.462399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573149.526305:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573149.526318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573149.526323:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573149.526333:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1365 00000100:00080000:0.0:1788573149.526344:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573149.526351:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573149.526390:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1c00 x1875453495488512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573149.526413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573149.526418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573149.526420:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573149.526424:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1365 00000100:00080000:0.0:1788573149.526428:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573149.526432:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573149.526442:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573149.526456:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495488640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573149.526462:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495488512:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573149.526467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573149.526470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573149.526471:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573149.526475:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1365 00000100:00080000:0.0:1788573149.526493:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573149.526497:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573149.526503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573149.526513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495488768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573149.526521:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573149.526523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573149.526526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573149.526528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573149.526531:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1365 00000100:00080000:0.0:1788573149.526533:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573149.526537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573149.526541:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573149.526549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495488896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573149.526561:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff93a2460d1c00 x1875453495488512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573149.526580:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1c00 x1875453495488512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573149.526592:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573149.526596:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573149.526600:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495488512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573149.526621:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573149.526640:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573149.526650:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff93a2460d1500 x1875453495488640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573149.526658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495488640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573149.526664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573149.526666:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573149.526668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573149.526679:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488768:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573149.526710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573149.526722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff93a2460d2680 x1875453495488768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573149.526736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495488768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573149.526745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573149.526748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573149.526750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495488768:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573149.526755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495488896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573149.526770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573149.526777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff93a2460d0380 x1875453495488896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573149.526786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495488896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573149.526795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573149.526798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573149.526801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495488896:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573150.486343:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573151.510424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573152.534371:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573153.558468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573154.582483:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573154.646372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573154.646403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573154.646410:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573154.646424:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1371 00000100:00080000:2.0:1788573154.646440:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573154.646452:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573154.646516:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495489024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573154.646559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573154.646566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573154.646569:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573154.646576:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1371 00000100:00080000:2.0:1788573154.646582:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573154.646589:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573154.646604:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573154.646627:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495489152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573154.646643:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495489024:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573154.646696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573154.646703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573154.646706:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573154.646714:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1371 00000100:00080000:2.0:1788573154.646721:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573154.646729:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573154.646739:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573154.646781:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573154.646788:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495489280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573154.646812:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573154.646818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573154.646822:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573154.646829:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1371 00000100:00080000:2.0:1788573154.646838:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573154.646851:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573154.646853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff93a249409180 x1875453495489024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573154.646862:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573154.646883:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495489408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573154.646905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495489024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573154.646922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573154.646929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573154.646935:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495489024:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573154.646967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573154.647004:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573154.647024:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff93a249408000 x1875453495489152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573154.647043:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495489152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573154.647057:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573154.647062:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573154.647068:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573154.647079:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489280:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573154.647112:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573154.647126:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff93a24940ad80 x1875453495489280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573154.647139:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495489280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573154.647167:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573154.647173:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573154.647178:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573154.647190:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495489408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573154.647265:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573154.647288:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff93a249408e00 x1875453495489408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573154.647306:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495489408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573154.647320:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573154.647325:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573154.647331:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495489408:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573155.606394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573156.630396:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573157.654386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573158.678379:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573159.702394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573159.766365:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.766377:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573159.766382:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.766394:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1376 00000100:00080000:2.0:1788573159.766404:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.766410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573159.766445:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495489536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.766470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.766474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573159.766488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.766519:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1376 00000100:00080000:2.0:1788573159.766523:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.766527:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573159.766546:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573159.766559:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495489536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573159.766560:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495489664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.766575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.766577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573159.766579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.766582:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1376 00000100:00080000:2.0:1788573159.766584:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.766586:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573159.766589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573159.766597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495489792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.766604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.766607:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573159.766609:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.766611:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1376 00000100:00080000:2.0:1788573159.766613:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.766615:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573159.766618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573159.766632:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495489920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573159.766646:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573159.766710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff93a24940b800 x1875453495489536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573159.766726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495489536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573159.766736:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573159.766740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573159.766743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495489536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573159.766770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573159.766790:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573159.766801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff93a249409c00 x1875453495489664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573159.766815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495489664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573159.766823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573159.766827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573159.766831:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573159.766838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489792:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573159.766861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573159.766873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff93a24940a680 x1875453495489792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573159.766882:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495489792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573159.766888:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573159.766891:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573159.766895:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495489792:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573159.766902:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495489920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573159.766923:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573159.766935:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff93a249408e00 x1875453495489920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573159.766953:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495489920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573159.766959:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573159.766961:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573159.766964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495489920:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573160.726425:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573161.750404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573162.774484:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573163.798460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573164.822474:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573164.886369:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573164.886382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573164.886386:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573164.886399:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1381 00000100:00080000:2.0:1788573164.886412:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573164.886420:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573164.886472:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495490048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573164.886507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573164.886511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573164.886512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573164.886516:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1381 00000100:00080000:2.0:1788573164.886519:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573164.886522:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573164.886533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573164.886545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495490176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573164.886552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573164.886555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573164.886557:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573164.886580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495490048:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573164.886581:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1381 00000100:00080000:2.0:1788573164.886588:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573164.886593:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573164.886597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573164.886605:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495490304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573164.886614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573164.886617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573164.886618:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573164.886621:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1381 00000100:00080000:2.0:1788573164.886624:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573164.886626:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573164.886629:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573164.886636:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495490432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573164.886673:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573164.886727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff93a24940ad80 x1875453495490048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573164.886749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495490048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573164.886760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573164.886765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573164.886770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495490048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573164.886779:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573164.886803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573164.886816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff93a249408000 x1875453495490176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573164.886829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495490176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573164.886852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573164.886856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573164.886859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573164.886888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490304:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573164.886909:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573164.886918:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff93a249409180 x1875453495490304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573164.886927:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495490304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573164.886933:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573164.886935:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573164.886937:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490304:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573164.886941:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495490432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573164.886951:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573164.886957:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff93a249408a80 x1875453495490432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573164.886964:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495490432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573164.886969:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573164.886970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573164.886977:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495490432:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573165.846480:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573166.870453:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573167.894423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573168.918400:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573169.942345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573170.006343:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573170.006352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573170.006356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573170.006364:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1386 00000100:00080000:2.0:1788573170.006373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573170.006379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573170.006426:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495490560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573170.006449:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573170.006451:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573170.006453:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573170.006456:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1386 00000100:00080000:2.0:1788573170.006458:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573170.006461:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573170.006469:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573170.006477:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495490688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573170.006484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573170.006487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573170.006488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573170.006491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1386 00000100:00080000:2.0:1788573170.006493:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573170.006496:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495490560:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573170.006497:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573170.006504:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573170.006513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495490816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573170.006521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573170.006524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573170.006525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573170.006533:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1386 00000100:00080000:2.0:1788573170.006544:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573170.006547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573170.006551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573170.006555:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573170.006561:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495490944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573170.006601:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff93a24940aa00 x1875453495490560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573170.006615:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495490560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573170.006623:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573170.006627:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573170.006631:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495490560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573170.006637:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573170.006653:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573170.006661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff93a249408700 x1875453495490688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573170.006669:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495490688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573170.006674:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573170.006677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573170.006679:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573170.006704:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490816:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573170.006721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573170.006746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff93a249409880 x1875453495490816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573170.006760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495490816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573170.006769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573170.006773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573170.006777:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495490816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573170.006781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495490944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573170.006798:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573170.006816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff93a249409500 x1875453495490944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573170.006824:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495490944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573170.006830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573170.006832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573170.006834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495490944:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573170.966404:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573171.990358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573173.014358:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573174.038376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573175.062425:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573175.126392:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.126402:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573175.126405:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.126414:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1391 00000100:00080000:0.0:1788573175.126424:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.126430:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573175.126464:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495491072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.126491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.126493:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573175.126495:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.126498:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1391 00000100:00080000:0.0:1788573175.126500:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.126502:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573175.126510:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573175.126518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495491200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.126526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.126529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573175.126531:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.126535:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1391 00000100:00080000:0.0:1788573175.126536:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.126538:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573175.126556:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573175.126563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495491328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.126575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573175.126577:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495491072:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573175.126579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573175.126583:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.126596:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1391 00000100:00080000:0.0:1788573175.126599:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.126602:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573175.126605:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573175.126614:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495491456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573175.126686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573175.126757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff93a2460d3480 x1875453495491072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573175.126782:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495491072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573175.126804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573175.126808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573175.126813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495491072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573175.126820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573175.126842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573175.126853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff93a2460d2a00 x1875453495491200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573175.126862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495491200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573175.126867:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573175.126869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573175.126872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573175.126905:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491328:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573175.126920:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573175.126928:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff93a2460d1180 x1875453495491328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573175.126935:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495491328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573175.126941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573175.126943:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573175.126946:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491328:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573175.126972:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495491456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573175.126989:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573175.126998:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff93a2460d2300 x1875453495491456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573175.127006:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495491456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573175.127012:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573175.127014:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573175.127016:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495491456:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573176.086382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573177.110457:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573178.134423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573179.158370:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573180.182425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573180.246306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.246316:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573180.246319:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.246327:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1396 00000100:00080000:0.0:1788573180.246337:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.246343:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573180.246382:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495491584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.246419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.246422:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573180.246437:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.246442:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1396 00000100:00080000:0.0:1788573180.246448:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.246453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573180.246463:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573180.246463:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495491584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573180.246486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495491712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.246499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.246502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573180.246504:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.246507:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1396 00000100:00080000:0.0:1788573180.246518:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573180.246523:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573180.246525:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573180.246532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573180.246543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495491840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.246550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.246553:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573180.246554:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.246558:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1396 00000100:00080000:0.0:1788573180.246560:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.246565:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573180.246566:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff93a2460d1f80 x1875453495491584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573180.246568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573180.246584:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495491584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573180.246587:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495491968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573180.246605:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573180.246610:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573180.246614:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495491584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573180.246634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573180.246651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573180.246660:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff93a2460d2300 x1875453495491712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573180.246676:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495491712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573180.246681:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573180.246683:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573180.246686:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573180.246689:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491840:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573180.246702:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573180.246709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff93a2460d1180 x1875453495491840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573180.246719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495491840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573180.246727:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573180.246730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573180.246734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495491840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573180.246740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495491968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573180.246759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573180.246771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff93a2460d2a00 x1875453495491968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573180.246903:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495491968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573180.246922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573180.246925:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573180.246929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495491968:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573181.206392:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573182.230436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573183.254385:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573184.278408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573185.302383:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573185.366369:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.366381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573185.366391:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.366403:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1401 00000100:00080000:2.0:1788573185.366413:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.366420:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573185.366453:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495492096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.366475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.366478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573185.366480:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.366483:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1401 00000100:00080000:2.0:1788573185.366485:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.366503:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573185.366512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573185.366522:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495492224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.366529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.366533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573185.366534:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.366538:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1401 00000100:00080000:2.0:1788573185.366541:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.366543:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573185.366547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573185.366556:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495492096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573185.366557:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495492352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.366583:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.366585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573185.366586:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.366591:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1401 00000100:00080000:2.0:1788573185.366593:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.366595:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573185.366613:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573185.366627:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495492480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573185.366671:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573185.366733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff93a24940b480 x1875453495492096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573185.366758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495492096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573185.366772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573185.366777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573185.366782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495492096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573185.366789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573185.366810:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573185.366820:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff93a24940bb80 x1875453495492224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573185.366829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495492224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573185.366834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573185.366837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573185.366839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573185.366878:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492352:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573185.366894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573185.366902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff93a24940b100 x1875453495492352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573185.366910:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495492352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573185.366915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573185.366918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573185.366920:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492352:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573185.366923:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495492480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573185.366933:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573185.366953:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff93a249409500 x1875453495492480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573185.366965:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495492480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573185.366974:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573185.366977:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573185.366981:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495492480:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573186.326490:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573187.350392:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573188.374404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573189.398362:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573190.422345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573190.486326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573190.486335:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573190.486339:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573190.486347:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1406 00000100:00080000:0.0:1788573190.486356:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573190.486362:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573190.486394:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495492608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573190.486416:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573190.486444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573190.486446:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573190.486450:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1406 00000100:00080000:0.0:1788573190.486454:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573190.486457:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573190.486465:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573190.486471:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495492608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573190.486480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495492736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573190.486489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573190.486513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573190.486515:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573190.486518:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1406 00000100:00080000:0.0:1788573190.486522:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573190.486525:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573190.486530:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573190.486538:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573190.486542:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495492864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573190.486551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573190.486555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573190.486556:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573190.486560:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1406 00000100:00080000:0.0:1788573190.486562:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573190.486564:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573190.486567:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573190.486576:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495492992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573190.486581:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff93a2460d3480 x1875453495492608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573190.486607:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495492608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573190.486615:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573190.486618:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573190.486622:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495492608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573190.486640:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573190.486657:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573190.486666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff93a2460d0380 x1875453495492736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573190.486673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495492736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573190.486679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573190.486681:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573190.486683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573190.486687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492864:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573190.486697:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573190.486703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff93a2460d2680 x1875453495492864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573190.486710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495492864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573190.486724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573190.486727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573190.486729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495492864:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573190.486732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495492992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573190.486746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573190.486752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff93a2460d1500 x1875453495492992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573190.486760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495492992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573190.486765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573190.486766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573190.486768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495492992:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573191.446380:0:20560:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573192.470384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573193.494399:0:17667:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 6/6 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788573193.494452:0:20560:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 6 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788573193.494465:0:20560:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff93a251f1d800 lock: ffff93a243216600/0x6d478a807bb11869 lrc: 2/0,0 mode: CR/CR res: [0x200001b71:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x977d9e7969c020be expref: -99 pid: 21304 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788573193.494476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000002:00100000:0.0:1788573193.494671:0:20560:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff93a24940a300 x1875453495433472/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 648/608 e 0 to 0 dl 1788572607 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:100 00000020:00100000:0.0:1788573193.494712:0:20560:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff93a251f1d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788573193.494717:0:20560:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff93a2460d1c00 x1875453495493248/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:100 00000100:00080000:0.0:1788573193.494740:0:20560:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff93a2460d1c00 x1875453495493248/t0(0) o35->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:100 00000100:00100000:0.0:1788573193.494749:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788573194.518407:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573194.518485:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573195.543008:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573195.543085:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573195.606342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573195.606351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573195.606355:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573195.606365:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1412 00000100:00080000:1.0:1788573195.606374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573195.606380:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573195.606413:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495493376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573195.606437:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573195.606441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573195.606448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573195.606453:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1412 00000100:00080000:1.0:1788573195.606457:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573195.606461:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573195.606471:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573195.606478:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495493376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573195.606485:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495493504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573195.606495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573195.606524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573195.606526:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573195.606531:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1412 00000100:00080000:1.0:1788573195.606535:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573195.606539:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573195.606544:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573195.606555:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495493632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573195.606566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573195.606569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573195.606571:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573195.606574:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1412 00000100:00080000:1.0:1788573195.606577:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573195.606581:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573195.606582:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495493504:192.168.202.102@tcp:38:kworker.0 00000008:00100000:1.0:1788573195.606585:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573195.606595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495493760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573195.606624:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495493632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573195.606656:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495493760:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573196.566477:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573196.566544:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573197.591678:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573197.592726:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573198.614553:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573198.614989:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:0.0:1788573198.678482:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788573198.678529:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573198.678546:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573198.678594:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573198.678605:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573198.678613:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573198.679281:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff93a258da3100 x1875453495493760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573198.679309:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495493760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573198.679337:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573198.679343:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573198.679347:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495493760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573198.679356:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff93a258da2a00 x1875453495493504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573198.679371:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495493504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573198.679380:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573198.679384:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573198.679388:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495493504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573198.679395:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff93a258da3480 x1875453495493632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573198.679407:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495493632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573198.679413:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573198.679416:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573198.679419:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495493632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573198.679428:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff93a258da0380 x1875453495493376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573198.679440:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495493376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573198.679454:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573198.679458:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573198.679461:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495493376:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788573199.639272:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573199.639305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788573200.662312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573200.662400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573200.726376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573200.726385:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573200.726388:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573200.726397:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1417 00000100:00080000:3.0:1788573200.726407:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573200.726414:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573200.726450:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495493888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573200.726483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573200.726486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573200.726487:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573200.726491:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1417 00000100:00080000:3.0:1788573200.726494:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573200.726497:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573200.726507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573200.726518:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495494016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573200.726527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573200.726531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573200.726559:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573200.726570:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1417 00000100:00100000:2.0:1788573200.726573:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495493888:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573200.726577:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573200.726587:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573200.726594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573200.726612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495494144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573200.726626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573200.726631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573200.726633:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573200.726638:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1417 00000100:00080000:3.0:1788573200.726660:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573200.726667:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573200.726673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573200.726684:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573200.726691:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495494272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573200.726761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff93a25064e300 x1875453495493888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.726794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495493888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.726808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.726814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573200.726818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495493888:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573200.726827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573200.726858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573200.726872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff93a25064c380 x1875453495494016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.726883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495494016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.726908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.726913:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573200.726917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573200.726964:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494144:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573200.726993:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573200.727006:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff93a25064f480 x1875453495494144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.727020:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495494144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.727031:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.727035:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573200.727039:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494144:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573200.727048:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495494272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573200.727073:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573200.727120:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff93a25064ce00 x1875453495494272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.727135:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495494272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.727144:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.727148:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573200.727161:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495494272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573201.686324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573201.686416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573202.710300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573202.710366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573203.734365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573203.734467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573204.758413:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573204.758413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573205.782341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573205.782426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573205.846326:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573205.846335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573205.846340:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573205.846351:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1422 00000100:00080000:2.0:1788573205.846362:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573205.846369:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573205.846410:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495494400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573205.846435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573205.846438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573205.846439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573205.846443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1422 00000100:00080000:2.0:1788573205.846446:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573205.846449:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573205.846457:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573205.846472:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495494528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573205.846485:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495494400:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573205.846487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573205.846509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573205.846512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573205.846528:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1422 00000100:00080000:2.0:1788573205.846542:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573205.846547:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573205.846554:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573205.846555:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573205.846573:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495494656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573205.846585:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573205.846588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573205.846590:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573205.846602:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1422 00000100:00100000:3.0:1788573205.846608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff93a249409880 x1875453495494400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573205.846610:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573205.846619:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573205.846628:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573205.846632:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495494400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573205.846642:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788573205.846644:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495494784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573205.846655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573205.846661:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495494400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573205.846684:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573205.846704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573205.846722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff93a249408700 x1875453495494528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573205.846738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495494528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573205.846747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573205.846751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573205.846755:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573205.846762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494656:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573205.846785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573205.846799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff93a24940aa00 x1875453495494656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573205.846810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495494656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573205.846816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573205.846819:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573205.846821:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495494656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573205.846827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495494784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573205.846858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573205.846871:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff93a249408a80 x1875453495494784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573205.846885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495494784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573205.846894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573205.846898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573205.846901:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495494784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573206.806364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573206.806410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573207.830327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573207.830380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573208.854307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573208.854429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573209.878324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573209.878400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573210.902316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573210.902383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573210.966357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.966367:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573210.966371:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.966380:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1427 00000100:00080000:1.0:1788573210.966389:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.966411:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573210.966440:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495494912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573210.966463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.966466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573210.966467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.966470:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1427 00000100:00080000:1.0:1788573210.966473:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.966476:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573210.966484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573210.966493:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495495040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573210.966498:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495494912:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573210.966504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.966508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573210.966509:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.966512:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1427 00000100:00080000:1.0:1788573210.966514:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.966516:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573210.966520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573210.966527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495495168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573210.966534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.966537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573210.966545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.966549:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1427 00000100:00080000:1.0:1788573210.966552:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.966555:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788573210.966559:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788573210.966562:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573210.966573:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495495296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573210.966599:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573210] req@ffff93a258da2300 x1875453495494912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573210.966615:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495494912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573210.966624:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573210.966627:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573210.966630:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495494912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573210.966648:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573210.966663:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573210.966671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573210] req@ffff93a258da0000 x1875453495495040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573210.966692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495495040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573210.966698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573210.966701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573210.966703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573210.966707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495168:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573210.966720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573210.966726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573210] req@ffff93a258da1180 x1875453495495168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573210.966734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495495168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573210.966739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573210.966741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573210.966743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495168:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573210.966746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495495296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573210.966756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573210.966762:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573210] req@ffff93a258da0380 x1875453495495296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573210.966769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495495296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573210.966784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573210.966786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573210.966788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495495296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573211.926334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573211.926404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573212.950328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573212.950390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573213.974367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573213.974470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573214.998350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573214.998427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573216.022307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573216.022421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573216.086390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.086399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573216.086402:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.086412:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1432 00000100:00080000:3.0:1788573216.086421:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.086427:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573216.086461:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495495424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.086492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.086495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573216.086496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.086499:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1432 00000100:00080000:3.0:1788573216.086501:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.086504:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573216.086512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573216.086520:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495495552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.086529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.086533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788573216.086534:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495495424:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573216.086550:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.086555:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1432 00000100:00080000:3.0:1788573216.086558:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.086561:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573216.086565:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573216.086574:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495495680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.086581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.086584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573216.086585:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.086588:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1432 00000100:00080000:3.0:1788573216.086590:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.086592:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573216.086595:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573216.086601:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573216.086605:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495495808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573216.086647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff93a25064f100 x1875453495495424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573216.086667:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495495424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573216.086675:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573216.086678:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573216.086681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495495424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573216.086699:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573216.086716:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573216.086725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff93a25064d500 x1875453495495552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573216.086732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495495552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573216.086738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573216.086740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573216.086743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573216.086746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495680:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573216.086757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573216.086771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff93a25064c700 x1875453495495680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573216.086779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495495680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573216.086784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573216.086786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573216.086801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495495680:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573216.086808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495495808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573216.086823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573216.086831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff93a25064ca80 x1875453495495808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573216.086838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495495808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573216.086844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573216.086846:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573216.086848:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495495808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573217.046302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573217.046401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573218.070308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573218.070392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573219.094306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573219.094354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573220.118355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573220.118393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573221.142330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573221.142392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573221.206325:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573221.206336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573221.206341:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573221.206352:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1437 00000100:00080000:3.0:1788573221.206364:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573221.206372:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573221.206411:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495495936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573221.206473:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573221.206477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573221.206479:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573221.206484:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1437 00000100:00080000:3.0:1788573221.206489:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573221.206495:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573221.206506:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573221.206516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495495936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573221.206525:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495496064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573221.206546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573221.206551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573221.206553:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573221.206558:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1437 00000100:00080000:3.0:1788573221.206570:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573221.206575:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:2.0:1788573221.206580:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788573221.206584:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573221.206611:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495496192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573221.206623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573221.206626:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff93a25064df80 x1875453495495936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.206629:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573221.206633:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573221.206638:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1437 00000100:00080000:3.0:1788573221.206652:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573221.206656:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495495936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.206661:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:2.0:1788573221.206672:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573221.206673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788573221.206678:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573221.206683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495495936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573221.206707:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495496320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573221.206731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573221.206752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573221.206761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff93a25064ca80 x1875453495496064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573221.206770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495496064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573221.206776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573221.206779:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573221.206781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573221.206785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496192:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573221.206797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573221.206803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff93a25064c700 x1875453495496192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573221.206810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495496192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573221.206825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573221.206827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573221.206830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573221.206834:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495496320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573221.206849:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573221.206861:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff93a25064d500 x1875453495496320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573221.206873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495496320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573221.206882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573221.206885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573221.206889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495496320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573222.166342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573222.166413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573223.190339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573223.190401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573224.214369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573224.214451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573225.238367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573225.238440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573226.262368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573226.262477:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573226.326376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.326387:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573226.326392:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.326403:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1442 00000100:00080000:3.0:1788573226.326415:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.326423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573226.326465:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495496448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.326494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.326498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573226.326499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.326502:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1442 00000100:00080000:3.0:1788573226.326504:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.326507:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573226.326517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573226.326526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495496576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.326546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.326570:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573226.326573:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.326579:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1442 00000100:00080000:3.0:1788573226.326585:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.326589:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573226.326594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573226.326621:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495496704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.326634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573226.326636:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495496448:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573226.326638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573226.326642:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.326651:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1442 00000100:00080000:3.0:1788573226.326655:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.326659:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573226.326662:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573226.326672:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495496832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573226.326727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573226.326775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff93a25064f100 x1875453495496448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573226.326794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495496448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573226.326825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573226.326830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573226.326835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495496448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573226.326843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573226.326869:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573226.326882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff93a25064ce00 x1875453495496576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573226.326895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495496576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573226.326903:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573226.326906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573226.326909:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573226.326935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496704:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573226.326955:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573226.326966:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff93a25064f480 x1875453495496704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573226.326977:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495496704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573226.326985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573226.326988:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573226.326991:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495496704:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573226.327005:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495496832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573226.327026:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573226.327037:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff93a25064c380 x1875453495496832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573226.327049:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495496832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573226.327068:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573226.327071:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573226.327074:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495496832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573227.286338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573227.286398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573228.310363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573228.310431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573229.334342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573229.334409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573230.358307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573230.358374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573231.382343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573231.382452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573231.446356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.446369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573231.446375:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.446389:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1447 00000100:00080000:3.0:1788573231.446403:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.446412:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573231.446459:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495496960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.446522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.446527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573231.446529:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.446535:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1447 00000100:00080000:3.0:1788573231.446541:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.446546:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573231.446560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573231.446573:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495496960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573231.446587:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495497088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.446602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.446607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573231.446609:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.446615:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1447 00000100:00080000:3.0:1788573231.446620:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.446624:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573231.446637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573231.446650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573231.446655:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495497216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.446673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.446678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573231.446680:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.446687:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1447 00000100:00100000:2.0:1788573231.446697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff93a25064e300 x1875453495496960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573231.446700:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.446716:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573231.446725:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573231.446726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495496960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573231.446743:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573231.446747:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495497344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573231.446751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573231.446764:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495496960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573231.446807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573231.446832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573231.446841:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff93a25064fb80 x1875453495497088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573231.446850:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495497088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573231.446855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573231.446858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573231.446860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573231.446864:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497216:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573231.446876:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573231.446882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff93a25064d180 x1875453495497216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573231.446890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495497216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573231.446895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573231.446897:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573231.446899:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573231.446903:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495497344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573231.446914:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573231.446920:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff93a25064c000 x1875453495497344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573231.446933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495497344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573231.446939:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573231.446941:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573231.446943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495497344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573232.406319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573232.406403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573233.430353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573233.430365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788573234.454365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573234.454486:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573235.478405:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573235.478454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573236.502326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573236.502390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573236.566362:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573236.566381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573236.566385:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573236.566393:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1453 00000100:00080000:2.0:1788573236.566412:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573236.566419:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573236.566454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495497472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573236.566478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573236.566483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573236.566485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573236.566491:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1453 00000100:00080000:2.0:1788573236.566503:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573236.566511:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573236.566516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495497472:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573236.566522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573236.566557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495497600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573236.566571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573236.566574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573236.566577:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573236.566582:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1453 00000100:00080000:2.0:1788573236.566587:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573236.566590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573236.566594:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573236.566601:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573236.566612:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495497728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573236.566621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573236.566623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573236.566625:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573236.566628:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1453 00000100:00080000:2.0:1788573236.566633:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573236.566633:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff93a249409180 x1875453495497472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573236.566641:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573236.566647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573236.566660:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495497472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573236.566661:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495497856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573236.566693:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573236.566697:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573236.566702:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495497472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573236.566729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573236.566752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573236.566763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff93a249408000 x1875453495497600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573236.566771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495497600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573236.566777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573236.566780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573236.566789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573236.566796:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497728:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573236.566812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573236.566818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff93a24940ad80 x1875453495497728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573236.566826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495497728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573236.566831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573236.566833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573236.566835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495497728:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573236.566838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495497856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573236.566847:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573236.566853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff93a249408e00 x1875453495497856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573236.566860:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495497856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573236.566864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573236.566866:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573236.566868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495497856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573237.526357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573237.526438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573238.550320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573238.550448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573239.574373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573239.574429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573240.598318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573240.598385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573241.622282:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573241.622370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573241.686360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573241.686372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573241.686377:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573241.686404:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1458 00000100:00080000:0.0:1788573241.686413:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573241.686419:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573241.686458:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495497984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573241.686478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573241.686481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573241.686483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573241.686486:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1458 00000100:00080000:0.0:1788573241.686488:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573241.686491:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573241.686497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573241.686507:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495498112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573241.686514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573241.686517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573241.686518:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573241.686521:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1458 00000100:00080000:0.0:1788573241.686524:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573241.686528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495497984:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573241.686533:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573241.686538:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573241.686548:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495498240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573241.686567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573241.686570:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573241.686571:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573241.686574:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1458 00000100:00080000:0.0:1788573241.686584:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573241.686588:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573241.686591:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573241.686595:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573241.686606:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495498368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573241.686642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff93a2460d0a80 x1875453495497984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573241.686657:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495497984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573241.686673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573241.686677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573241.686680:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495497984:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573241.686686:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573241.686709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573241.686718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff93a2460d0000 x1875453495498112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573241.686727:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495498112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573241.686733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573241.686735:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573241.686737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573241.686755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498240:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573241.686768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573241.686775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff93a2460d3b80 x1875453495498240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573241.686782:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495498240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573241.686797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573241.686800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573241.686803:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498240:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573241.686807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495498368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573241.686820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573241.686827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff93a2460d0700 x1875453495498368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573241.686835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495498368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573241.686840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573241.686842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573241.686844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495498368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573242.646345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573242.646394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573243.670316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573243.670386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573244.694348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573244.694400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573245.718348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573245.718430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573246.742342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573246.742408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573246.806341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.806353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573246.806357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.806368:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1463 00000100:00080000:3.0:1788573246.806380:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.806388:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573246.806427:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495498496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.806460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.806484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.806488:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.806494:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1463 00000100:00080000:3.0:1788573246.806499:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573246.806506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495498496:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573246.806507:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573246.806523:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573246.806542:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495498624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.806556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.806560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.806563:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.806568:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1463 00000100:00080000:3.0:1788573246.806572:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573246.806575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573246.806580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573246.806588:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573246.806603:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495498752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.806616:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.806620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.806622:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573246.806626:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff93a25064d880 x1875453495498496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573246.806631:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1463 00000100:00080000:3.0:1788573246.806640:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.806646:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573246.806650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495498496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573246.806665:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573246.806665:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788573246.806672:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573246.806676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495498496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573246.806699:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573246.806701:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495498880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573246.806720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573246.806733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff93a25064e680 x1875453495498624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573246.806743:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495498624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573246.806749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573246.806752:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573246.806754:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573246.806765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498752:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573246.806780:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573246.806787:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff93a25064ed80 x1875453495498752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573246.806795:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495498752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573246.806800:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573246.806802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573246.806804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495498752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573246.806813:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495498880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573246.806830:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573246.806841:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff93a25064dc00 x1875453495498880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573246.806854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495498880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573246.806863:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573246.806867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573246.806870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495498880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573247.766350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573247.766412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573248.790360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573248.790406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573249.814352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573249.814420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573250.838364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573250.838448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573251.862326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573251.862378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573251.926306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573251.926316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573251.926322:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573251.926332:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1468 00000100:00080000:3.0:1788573251.926342:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573251.926350:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573251.926394:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495499008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573251.926421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573251.926425:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573251.926427:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573251.926432:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1468 00000100:00080000:3.0:1788573251.926436:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573251.926441:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573251.926451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573251.926460:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495499008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573251.926469:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495499136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573251.926481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573251.926486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573251.926488:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573251.926493:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1468 00000100:00080000:3.0:1788573251.926511:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573251.926525:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573251.926530:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573251.926539:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573251.926555:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495499264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573251.926567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573251.926571:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573251.926573:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573251.926577:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff93a25064ea00 x1875453495499008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573251.926581:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1468 00000100:00080000:3.0:1788573251.926589:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573251.926596:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573251.926602:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495499008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573251.926602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788573251.926615:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.926618:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573251.926622:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495499008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573251.926632:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495499392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573251.926644:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573251.926665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573251.926675:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff93a25064dc00 x1875453495499136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.926683:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495499136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.926689:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.926691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573251.926694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573251.926697:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499264:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573251.926709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573251.926714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff93a25064ed80 x1875453495499264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.926729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495499264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.926734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.926736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573251.926738:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499264:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573251.926748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495499392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573251.926761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573251.926767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff93a25064e680 x1875453495499392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.926775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495499392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.926780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.926782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573251.926784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495499392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573252.886320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573252.886398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573253.910302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573253.910373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573254.934313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573254.934375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573255.958310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573255.958447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573256.982377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573256.982445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573257.046347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.046358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573257.046362:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.046374:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1473 00000100:00080000:3.0:1788573257.046386:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.046393:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573257.046437:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495499520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573257.046467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.046489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573257.046490:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.046494:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1473 00000100:00080000:3.0:1788573257.046498:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.046502:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573257.046512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573257.046524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495499648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573257.046530:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495499520:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573257.046535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.046539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573257.046540:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.046545:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1473 00000100:00080000:3.0:1788573257.046547:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.046549:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573257.046552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573257.046562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495499776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573257.046570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.046573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573257.046575:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.046577:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1473 00000100:00080000:3.0:1788573257.046594:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.046596:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573257.046608:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573257.046610:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573257.046625:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495499904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573257.046670:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff93a25064d880 x1875453495499520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573257.046694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495499520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573257.046706:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573257.046712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573257.046716:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495499520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573257.046749:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573257.046778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573257.046793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff93a25064c000 x1875453495499648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573257.046807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495499648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573257.046817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573257.046821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573257.046825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573257.046832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499776:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573257.046859:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573257.046873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff93a25064d180 x1875453495499776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573257.046887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495499776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573257.046911:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573257.046915:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573257.046919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495499776:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573257.046927:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495499904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573257.046964:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573257.046978:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff93a25064fb80 x1875453495499904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573257.046992:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495499904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573257.047002:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573257.047006:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573257.047010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495499904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573258.006340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573258.006375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573259.030346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573259.030420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573260.054341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573260.054402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573261.078379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573261.078720:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573262.103313:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573262.103513:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788573262.166351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.166361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573262.166365:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.166374:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1478 00000100:00080000:3.0:1788573262.166382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.166388:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573262.166427:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495500032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573262.166453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.166455:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573262.166457:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.166460:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1478 00000100:00080000:3.0:1788573262.166463:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.166466:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573262.166474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573262.166485:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495500160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573262.166492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.166494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573262.166495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.166498:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1478 00000100:00080000:3.0:1788573262.166514:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573262.166519:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495500032:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573262.166520:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573262.166538:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573262.166549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495500288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573262.166559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.166562:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573262.166563:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.166566:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1478 00000100:00080000:3.0:1788573262.166571:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.166574:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573262.166577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573262.166585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495500416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573262.166653:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573262.166687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573262.166705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495500416:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573263.126457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573263.126635:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573264.150461:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573264.150616:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573265.174451:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573265.174726:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788573265.238502:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788573265.238547:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573265.238564:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573265.238600:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573265.238610:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573265.238620:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:2.0:1788573265.238646:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff93a25064f480 x1875453495500288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573265.238674:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495500288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573265.238687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573265.238693:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573265.238698:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573265.238708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff93a25064ce00 x1875453495500416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573265.238723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495500416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573265.238731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573265.238735:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573265.238739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495500416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573265.238746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff93a25064e300 x1875453495500032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573265.238759:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495500032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573265.238766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573265.238787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573265.238791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495500032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573265.238798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff93a25064c380 x1875453495500160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573265.238813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495500160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573265.238821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573265.238825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573265.238827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500160:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573266.198776:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573266.198928:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573267.222336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573267.222383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573267.286393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.286404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573267.286408:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.286419:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1483 00000100:00080000:0.0:1788573267.286429:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.286436:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573267.286473:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495500544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573267.286507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.286511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573267.286513:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.286518:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1483 00000100:00080000:0.0:1788573267.286523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.286527:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573267.286538:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573267.286554:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495500672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573267.286562:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495500544:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573267.286592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.286596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573267.286598:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.286604:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1483 00000100:00080000:0.0:1788573267.286609:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.286613:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573267.286620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573267.286637:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495500800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573267.286637:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573267.286652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.286656:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573267.286659:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.286663:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1483 00000100:00080000:0.0:1788573267.286667:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.286671:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573267.286677:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573267.286687:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff93a2460d0700 x1875453495500544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573267.286692:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495500928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573267.286724:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495500544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573267.286738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573267.286743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573267.286748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495500544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573267.286797:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573267.286825:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573267.286864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff93a2460d3b80 x1875453495500672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573267.286877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495500672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573267.286887:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573267.286891:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573267.286895:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573267.286901:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500800:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573267.286936:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573267.286950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff93a2460d0000 x1875453495500800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573267.286963:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495500800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573267.286971:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573267.286975:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573267.286978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495500800:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573267.286984:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495500928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573267.287006:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573267.287017:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff93a2460d0a80 x1875453495500928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573267.287036:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495500928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573267.287052:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573267.287056:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573267.287059:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495500928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573268.246351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573268.246409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573269.270336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573269.270385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573270.294362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573270.294462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573271.318341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573271.318379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573272.342338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573272.342407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573272.406318:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.406327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573272.406330:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.406339:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1488 00000100:00080000:2.0:1788573272.406347:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.406352:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573272.406388:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495501056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.406418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.406422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573272.406424:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.406428:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1488 00000100:00080000:2.0:1788573272.406439:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.406443:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573272.406452:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573272.406458:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495501056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573272.406466:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495501184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.406503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.406507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573272.406509:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.406514:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1488 00000100:00080000:2.0:1788573272.406519:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.406523:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788573272.406529:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:2.0:1788573272.406532:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573272.406550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495501312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.406560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.406565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573272.406567:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.406572:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1488 00000100:00100000:3.0:1788573272.406577:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff93a24940a680 x1875453495501056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573272.406579:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.406586:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573272.406594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573272.406612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495501440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573272.406616:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495501056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573272.406630:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573272.406638:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573272.406644:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495501056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573272.406668:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573272.406688:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573272.406699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff93a249409c00 x1875453495501184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573272.406710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495501184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573272.406719:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573272.406723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573272.406726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573272.406733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501312:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573272.406757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573272.406783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff93a24940b800 x1875453495501312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573272.406833:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495501312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573272.406855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573272.406858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573272.406863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501312:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573272.406869:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495501440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573272.406887:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573272.406895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff93a249408e00 x1875453495501440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573272.406903:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495501440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573272.406908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573272.406910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573272.406912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495501440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573273.366310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573273.366383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573274.390345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573274.390411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573275.414294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573275.414381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573276.438396:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573276.438452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573277.462332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573277.462381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573277.526378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573277.526390:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573277.526394:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573277.526405:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1493 00000100:00080000:3.0:1788573277.526415:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573277.526423:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573277.526471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495501568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573277.526497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573277.526500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573277.526501:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573277.526505:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1493 00000100:00080000:3.0:1788573277.526508:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573277.526512:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573277.526520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573277.526528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495501568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573277.526534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495501696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573277.526546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573277.526550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573277.526552:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573277.526556:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1493 00000100:00080000:3.0:1788573277.526560:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573277.526563:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573277.526569:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573277.526581:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495501824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573277.526591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573277.526595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573277.526597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573277.526600:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573277.526609:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1493 00000100:00080000:3.0:1788573277.526614:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573277.526619:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573277.526625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573277.526636:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495501952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573277.526650:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff93a25064f100 x1875453495501568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573277.526679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495501568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573277.526689:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573277.526693:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573277.526697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495501568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573277.526718:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573277.526746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573277.526754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff93a25064d500 x1875453495501696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573277.526762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495501696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573277.526767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573277.526770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573277.526773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573277.526778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501824:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573277.526797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573277.526807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff93a25064c700 x1875453495501824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573277.526818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495501824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573277.526831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573277.526835:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573277.526838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495501824:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573277.526843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495501952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573277.526866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573277.526877:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff93a25064ca80 x1875453495501952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573277.526888:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495501952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573277.526896:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573277.526899:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573277.526902:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495501952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573278.486303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573278.486336:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573279.510373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573279.510488:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573280.534294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573280.534413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573281.558368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573281.558440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573282.582300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573282.582375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573282.646371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573282.646378:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573282.646386:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573282.646394:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1499 00000100:00080000:2.0:1788573282.646402:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573282.646408:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573282.646437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495502080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573282.646462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573282.646477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573282.646478:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573282.646481:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1499 00000100:00080000:2.0:1788573282.646484:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573282.646486:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573282.646498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573282.646507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495502208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573282.646513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573282.646516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573282.646517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573282.646520:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1499 00000100:00080000:2.0:1788573282.646522:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573282.646524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573282.646526:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573282.646532:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495502336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573282.646539:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495502080:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573282.646541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573282.646547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573282.646549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573282.646551:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1499 00000100:00080000:2.0:1788573282.646553:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573282.646555:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573282.646563:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573282.646571:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495502464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573282.646597:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573282.646646:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff93a24940ad80 x1875453495502080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573282.646675:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495502080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573282.646688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573282.646691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573282.646695:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495502080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573282.646701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573282.646717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573282.646725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff93a249408000 x1875453495502208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573282.646742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495502208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573282.646748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573282.646750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573282.646752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573282.646774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502336:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573282.646794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573282.646805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff93a249409180 x1875453495502336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573282.646816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495502336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573282.646820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573282.646822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573282.646824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502336:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573282.646828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495502464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573282.646840:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573282.646846:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff93a249408a80 x1875453495502464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573282.646852:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495502464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573282.646864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573282.646866:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573282.646867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495502464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573283.606358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573283.606424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573284.630313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573284.630389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573285.654331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573285.654392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573286.678360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573286.678431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573287.702320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573287.702411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573287.766348:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573287.766357:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573287.766360:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573287.766369:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1504 00000100:00080000:1.0:1788573287.766377:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573287.766384:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573287.766422:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495502592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573287.766450:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573287.766453:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573287.766454:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573287.766457:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1504 00000100:00080000:1.0:1788573287.766460:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573287.766463:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573287.766483:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573287.766492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495502720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573287.766500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573287.766503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573287.766504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495502592:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573287.766505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573287.766512:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1504 00000100:00080000:1.0:1788573287.766516:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573287.766518:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573287.766521:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573287.766527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495502848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573287.766533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573287.766535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573287.766536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573287.766538:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1504 00000100:00080000:1.0:1788573287.766540:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573287.766542:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573287.766544:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573287.766550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495502976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573287.766584:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573287.766666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff93a258da3480 x1875453495502592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573287.766684:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495502592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573287.766695:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573287.766699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573287.766703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495502592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573287.766724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573287.766756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573287.766768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff93a258da2a00 x1875453495502720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573287.766781:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495502720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573287.766798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573287.766802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573287.766805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573287.766811:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502848:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573287.766834:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573287.766845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff93a258da3100 x1875453495502848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573287.766856:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495502848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573287.766864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573287.766867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573287.766870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495502848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573287.766875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495502976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573287.766893:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573287.766903:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff93a258da1880 x1875453495502976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573287.766923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495502976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573287.766931:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573287.766933:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573287.766943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495502976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573288.726324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573288.726386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573289.750317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573289.750383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573290.774330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573290.774430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573291.798404:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573291.798473:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573292.822337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573292.822385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573292.886405:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573292.886414:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573292.886419:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573292.886430:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1509 00000100:00080000:2.0:1788573292.886443:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573292.886450:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573292.886495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495503104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573292.886520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573292.886523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573292.886525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573292.886528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1509 00000100:00080000:2.0:1788573292.886531:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573292.886533:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573292.886553:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573292.886566:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495503232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573292.886574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573292.886577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573292.886579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573292.886582:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1509 00000100:00080000:2.0:1788573292.886603:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573292.886615:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573292.886618:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495503104:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573292.886624:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573292.886647:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495503360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573292.886663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573292.886668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573292.886670:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573292.886677:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1509 00000100:00080000:2.0:1788573292.886685:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573292.886691:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573292.886696:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573292.886698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573292.886721:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495503488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573292.886769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff93a24940aa00 x1875453495503104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573292.886797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495503104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573292.886806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573292.886810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573292.886813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495503104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573292.886819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573292.886837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573292.886845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff93a249408700 x1875453495503232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573292.886853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495503232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573292.886858:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573292.886860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573292.886875:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573292.886898:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503360:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573292.886921:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573292.886933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff93a249409880 x1875453495503360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573292.886946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495503360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573292.886955:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573292.886957:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573292.886960:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503360:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573292.886964:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495503488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573292.886980:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573292.886987:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff93a249409500 x1875453495503488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573292.886995:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495503488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573292.887000:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573292.887002:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573292.887003:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495503488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573293.846321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573293.846413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573294.870318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573294.870385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573295.894327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573295.894438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573296.918337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573296.918417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573297.942314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573297.942364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573298.006346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573298.006359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573298.006364:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573298.006375:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1514 00000100:00080000:2.0:1788573298.006388:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573298.006397:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573298.006443:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495503616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573298.006471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573298.006516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573298.006518:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573298.006522:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1514 00000100:00080000:2.0:1788573298.006526:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573298.006531:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573298.006545:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573298.006547:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495503616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573298.006562:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495503744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573298.006582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573298.006586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573298.006587:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573298.006594:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1514 00000100:00080000:2.0:1788573298.006598:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573298.006604:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573298.006610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573298.006623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495503872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573298.006632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573298.006635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573298.006637:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573298.006643:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1514 00000800:00000100:3.0:1788573298.006648:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573298.006651:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573298.006657:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573298.006668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573298.006681:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495504000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573298.006716:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff93a24940b100 x1875453495503616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573298.006738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495503616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573298.006751:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573298.006755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573298.006759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495503616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573298.006785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573298.006811:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573298.006824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff93a24940bb80 x1875453495503744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573298.006837:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495503744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573298.006845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573298.006849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573298.006853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573298.006860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503872:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573298.006899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573298.006910:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff93a24940b480 x1875453495503872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573298.006923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495503872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573298.006931:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573298.006935:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573298.006938:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495503872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573298.006944:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495504000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573298.006965:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573298.006976:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff93a249409500 x1875453495504000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573298.006987:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495504000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573298.006995:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573298.006998:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573298.007002:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495504000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573298.966336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573298.966395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573299.990303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573299.990351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573301.014326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573301.014378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573302.038290:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573302.038442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573303.062333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573303.062396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573303.126343:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573303.126352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573303.126355:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573303.126363:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1519 00000100:00080000:0.0:1788573303.126390:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573303.126396:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573303.126467:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495504128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573303.126490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573303.126493:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573303.126494:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573303.126497:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1519 00000100:00080000:0.0:1788573303.126500:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573303.126503:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573303.126511:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573303.126519:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495504256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573303.126526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573303.126529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573303.126530:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573303.126532:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1519 00000100:00080000:0.0:1788573303.126534:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573303.126536:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573303.126540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573303.126551:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495504128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573303.126554:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495504384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573303.126564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573303.126573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573303.126575:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573303.126578:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1519 00000100:00080000:0.0:1788573303.126581:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573303.126584:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573303.126588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573303.126598:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495504512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573303.126641:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573303.126693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff93a2460d1500 x1875453495504128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573303.126714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495504128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573303.126726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573303.126730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573303.126734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495504128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573303.126743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573303.126769:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573303.126784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff93a2460d2680 x1875453495504256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573303.126797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495504256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573303.126806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573303.126810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573303.126813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573303.126837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504384:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573303.126860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573303.126873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff93a2460d0380 x1875453495504384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573303.126902:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495504384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573303.126910:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573303.126913:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573303.126917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504384:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573303.126924:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495504512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573303.126946:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573303.126958:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff93a2460d3480 x1875453495504512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573303.126971:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495504512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573303.126979:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573303.126982:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573303.126986:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495504512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573304.086307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573304.086386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573305.110339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573305.110409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573306.134321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573306.134377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573307.158348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573307.158413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573308.182323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573308.182390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573308.246346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573308.246369:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573308.246374:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573308.246384:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1524 00000100:00080000:2.0:1788573308.246394:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573308.246403:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573308.246438:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495504640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573308.246466:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573308.246470:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573308.246472:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573308.246477:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1524 00000100:00080000:2.0:1788573308.246480:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573308.246485:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573308.246494:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573308.246509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495504768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573308.246511:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495504640:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573308.246528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573308.246531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573308.246533:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573308.246537:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1524 00000100:00080000:2.0:1788573308.246565:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573308.246571:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573308.246578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573308.246580:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573308.246593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495504896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573308.246605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573308.246608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573308.246610:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573308.246615:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1524 00000100:00080000:2.0:1788573308.246619:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573308.246622:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573308.246627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573308.246638:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495505024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573308.246642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff93a249409880 x1875453495504640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573308.246666:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495504640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573308.246674:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573308.246678:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573308.246681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495504640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573308.246700:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573308.246719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573308.246728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff93a249408700 x1875453495504768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573308.246736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495504768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573308.246742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573308.246744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573308.246746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573308.246750:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504896:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573308.246762:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573308.246768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff93a24940aa00 x1875453495504896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573308.246776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495504896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573308.246789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573308.246792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573308.246794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495504896:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573308.246798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495505024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573308.246812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573308.246819:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff93a249408a80 x1875453495505024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573308.246826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495505024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573308.246831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573308.246833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573308.246835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495505024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573309.206341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573309.206418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573310.230300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573310.230419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573311.254361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573311.254425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573312.278322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573312.278393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573313.302359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573313.302436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573313.366366:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573313.366374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573313.366378:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573313.366387:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1529 00000100:00080000:0.0:1788573313.366395:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573313.366401:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573313.366431:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495505152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573313.366469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573313.366472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573313.366473:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573313.366477:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1529 00000100:00080000:0.0:1788573313.366479:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573313.366482:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573313.366490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573313.366499:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495505280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573313.366506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573313.366509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573313.366510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573313.366513:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1529 00000100:00080000:0.0:1788573313.366516:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573313.366518:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573313.366522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573313.366527:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495505152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573313.366531:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495505408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573313.366543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573313.366545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573313.366546:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573313.366548:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1529 00000100:00080000:0.0:1788573313.366556:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573313.366559:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573313.366562:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573313.366572:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495505536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573313.366601:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573313.366661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff93a2460d2a00 x1875453495505152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573313.366682:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495505152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573313.366700:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573313.366705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573313.366710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495505152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573313.366718:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573313.366745:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573313.366760:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff93a2460d1180 x1875453495505280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573313.366774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495505280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573313.366794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573313.366798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573313.366803:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573313.366828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505408:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573313.366853:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573313.366866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff93a2460d2300 x1875453495505408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573313.366879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495505408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573313.366888:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573313.366891:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573313.366895:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573313.366910:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495505536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573313.366933:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573313.366945:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff93a2460d1f80 x1875453495505536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573313.366958:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495505536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573313.366966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573313.366969:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573313.366973:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495505536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573314.326342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573314.326415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573315.350340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573315.350410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573316.374352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573316.374413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573317.398333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573317.398387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573318.422344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573318.422402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573318.486315:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573318.486324:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573318.486328:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573318.486336:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1534 00000100:00080000:0.0:1788573318.486344:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573318.486350:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573318.486394:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495505664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573318.486415:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573318.486418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573318.486419:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573318.486422:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1534 00000100:00080000:0.0:1788573318.486425:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573318.486427:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573318.486434:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573318.486459:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495505792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573318.486467:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495505664:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573318.486470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573318.486477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573318.486479:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573318.486483:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1534 00000100:00080000:0.0:1788573318.486486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573318.486488:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573318.486492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573318.486500:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495505920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573318.486516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573318.486518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573318.486520:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573318.486523:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1534 00000100:00080000:0.0:1788573318.486527:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573318.486529:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573318.486532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573318.486538:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573318.486542:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495506048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573318.486581:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff93a2460d1f80 x1875453495505664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573318.486600:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495505664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573318.486610:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573318.486614:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573318.486618:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495505664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573318.486647:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573318.486670:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573318.486683:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff93a2460d2300 x1875453495505792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573318.486695:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495505792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573318.486703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573318.486707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573318.486710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573318.486717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505920:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573318.486734:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573318.486745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff93a2460d1180 x1875453495505920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573318.486756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495505920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573318.486765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573318.486768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573318.486772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495505920:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573318.486778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495506048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573318.486795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573318.486805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff93a2460d2a00 x1875453495506048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573318.486825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495506048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573318.486833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573318.486837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573318.486840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495506048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573319.446314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573319.446383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573320.470303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573320.470386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573321.494305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573321.494395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573322.518336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573322.518398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573323.542334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573323.542409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573323.606362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.606371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573323.606374:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.606383:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1540 00000100:00080000:2.0:1788573323.606392:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.606398:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573323.606430:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495506176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.606453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.606476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573323.606478:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.606483:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1540 00000100:00080000:2.0:1788573323.606487:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.606491:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573323.606497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495506176:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573323.606501:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573323.606513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495506304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.606524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.606527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573323.606529:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.606533:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1540 00000100:00080000:2.0:1788573323.606536:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.606539:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573323.606543:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573323.606552:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495506432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.606561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.606565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788573323.606565:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573323.606567:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.606570:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1540 00000100:00080000:2.0:1788573323.606573:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.606576:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573323.606587:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573323.606598:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495506560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573323.606618:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff93a249409180 x1875453495506176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573323.606651:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495506176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573323.606662:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573323.606666:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573323.606669:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495506176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573323.606690:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573323.606709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573323.606719:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff93a249408000 x1875453495506304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573323.606733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495506304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573323.606739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573323.606741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573323.606743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573323.606747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506432:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573323.606770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573323.606777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff93a24940ad80 x1875453495506432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573323.606785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495506432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573323.606790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573323.606792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573323.606794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506432:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573323.606797:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495506560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573323.606807:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573323.606813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff93a249408e00 x1875453495506560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573323.606820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495506560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573323.606833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573323.606842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573323.606845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495506560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573324.566336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573324.566403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573325.590339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573325.590396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573326.614339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573326.614411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573327.638364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573327.638454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573328.662391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573328.663167:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573328.726335:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573328.726344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573328.726348:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573328.726357:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1545 00000100:00080000:2.0:1788573328.726366:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573328.726371:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573328.726403:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495506688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573328.726426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573328.726428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573328.726430:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573328.726433:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1545 00000100:00080000:2.0:1788573328.726435:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573328.726438:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573328.726445:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573328.726455:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495506816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573328.726462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573328.726465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573328.726466:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573328.726469:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1545 00000100:00100000:3.0:1788573328.726474:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495506688:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573328.726484:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573328.726489:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573328.726493:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573328.726503:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495506944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573328.726511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573328.726514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573328.726515:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573328.726518:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1545 00000100:00080000:2.0:1788573328.726528:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573328.726544:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573328.726548:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573328.726557:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495507072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573328.726575:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573328.726620:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573328.726639:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495507072:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573329.686707:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573329.686844:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788573330.711621:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573330.714339:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573331.734328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573331.734471:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573331.798468:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788573331.798511:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573331.798526:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573331.798561:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573331.798570:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573331.798578:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573331.798617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff93a24940a680 x1875453495506944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573331.798643:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495506944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573331.798656:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573331.798661:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573331.798665:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573331.798673:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff93a249408e00 x1875453495507072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573331.798703:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495507072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573331.798713:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573331.798717:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573331.798720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495507072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573331.798727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff93a24940b800 x1875453495506688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573331.798739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495506688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573331.798746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573331.798749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573331.798752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495506688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573331.798759:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff93a249409c00 x1875453495506816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573331.798770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495506816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573331.798778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573331.798781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573331.798784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495506816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573332.758504:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573332.758699:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788573333.782376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573333.782416:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:2.0:1788573333.846321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573333.846331:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573333.846335:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573333.846347:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1550 00000100:00080000:2.0:1788573333.846357:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573333.846364:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573333.846414:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495507200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573333.846443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573333.846447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573333.846449:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573333.846454:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1550 00000100:00080000:2.0:1788573333.846458:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573333.846463:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573333.846473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573333.846486:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495507200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573333.846489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495507328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573333.846503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573333.846507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573333.846509:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573333.846513:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1550 00000100:00080000:2.0:1788573333.846517:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573333.846521:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573333.846526:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573333.846537:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495507456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573333.846547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573333.846552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788573333.846553:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573333.846556:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573333.846570:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1550 00000100:00080000:2.0:1788573333.846576:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573333.846580:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573333.846586:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573333.846598:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495507584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573333.846609:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff93a24940ad80 x1875453495507200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573333.846637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495507200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573333.846649:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573333.846680:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573333.846685:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495507200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573333.846716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573333.846743:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573333.846758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff93a249408000 x1875453495507328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573333.846777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495507328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573333.846786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573333.846790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573333.846794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573333.846800:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507456:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573333.846823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573333.846834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff93a249409180 x1875453495507456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573333.846856:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495507456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573333.846865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573333.846869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573333.846872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507456:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573333.846878:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495507584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573333.846899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573333.846913:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff93a249408a80 x1875453495507584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573333.846925:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495507584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573333.846932:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573333.846936:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573333.846939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495507584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573334.806342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573334.806386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573335.830340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573335.830393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573336.854324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573336.854386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573337.878330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573337.878394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573338.902334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573338.902392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573338.966359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573338.966368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573338.966389:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573338.966400:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1555 00000100:00080000:1.0:1788573338.966411:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573338.966419:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573338.966453:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495507712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573338.966492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573338.966496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573338.966498:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573338.966503:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1555 00000100:00080000:1.0:1788573338.966507:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573338.966512:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573338.966523:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573338.966535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495507840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573338.966539:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495507712:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573338.966547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573338.966566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573338.966567:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573338.966572:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1555 00000100:00080000:1.0:1788573338.966576:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573338.966580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573338.966584:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573338.966595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495507968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573338.966604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573338.966605:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788573338.966611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573338.966614:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573338.966620:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1555 00000100:00080000:1.0:1788573338.966627:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573338.966643:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573338.966646:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff93a258da2680 x1875453495507712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573338.966650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573338.966666:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495508096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573338.966673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495507712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573338.966684:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573338.966687:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573338.966691:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495507712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573338.966710:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573338.966728:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573338.966736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff93a258da1c00 x1875453495507840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573338.966745:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495507840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573338.966762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573338.966765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573338.966769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573338.966776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507968:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573338.966799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573338.966811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff93a258da0700 x1875453495507968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573338.966820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495507968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573338.966838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573338.966842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573338.966845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495507968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573338.966852:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495508096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573338.966874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573338.966886:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff93a258da0a80 x1875453495508096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573338.966895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495508096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573338.966901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573338.966903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573338.966906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495508096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573339.926335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573339.926420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573340.950312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573340.950358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573341.974374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573341.974441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573342.998386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573342.998445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573344.022345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573344.022467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573344.086297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.086309:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573344.086314:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.086325:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1560 00000100:00080000:1.0:1788573344.086338:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.086346:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573344.086390:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495508224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.086436:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.086440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573344.086441:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.086445:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1560 00000100:00080000:1.0:1788573344.086448:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.086451:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573344.086461:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573344.086472:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495508352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.086480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.086483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573344.086484:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.086488:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1560 00000100:00080000:1.0:1788573344.086491:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.086496:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573344.086496:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495508224:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788573344.086504:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573344.086513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495508480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.086521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.086524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573344.086526:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.086528:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1560 00000100:00080000:1.0:1788573344.086539:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.086543:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573344.086547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573344.086557:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495508608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573344.086574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573344.086621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff93a258da1f80 x1875453495508224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573344.086637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495508224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573344.086652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573344.086656:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573344.086659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495508224:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573344.086664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573344.086683:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573344.086692:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff93a258da0e00 x1875453495508352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573344.086700:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495508352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573344.086706:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573344.086708:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573344.086710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573344.086733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508480:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573344.086745:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573344.086751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff93a258da3800 x1875453495508480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573344.086758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495508480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573344.086763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573344.086765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573344.086767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508480:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573344.086775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495508608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573344.086798:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573344.086806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff93a258da0a80 x1875453495508608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573344.086814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495508608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573344.086819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573344.086821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573344.086823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495508608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573345.046331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573345.046398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573346.070308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573346.070360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788573347.094411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573347.094440:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573348.118344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573348.118410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788573349.142351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573349.142358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:1.0:1788573349.206324:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573349.206336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573349.206340:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573349.206351:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1565 00000100:00080000:1.0:1788573349.206363:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573349.206371:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573349.206410:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495508736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573349.206433:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573349.206436:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573349.206438:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573349.206441:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1565 00000100:00080000:1.0:1788573349.206444:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573349.206447:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573349.206454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573349.206462:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495508864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573349.206470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573349.206487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573349.206489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573349.206489:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495508736:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573349.206498:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1565 00000100:00080000:1.0:1788573349.206501:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573349.206503:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573349.206507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573349.206517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495508992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573349.206526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573349.206528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573349.206529:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573349.206531:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1565 00000100:00080000:1.0:1788573349.206534:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573349.206536:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573349.206539:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573349.206547:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495509120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573349.206555:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573349.206610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff93a258da0700 x1875453495508736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573349.206630:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495508736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573349.206641:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573349.206645:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573349.206649:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495508736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573349.206665:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573349.206690:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573349.206704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff93a258da1c00 x1875453495508864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573349.206717:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495508864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573349.206736:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573349.206739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573349.206742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573349.206768:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508992:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573349.206792:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573349.206804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff93a258da2680 x1875453495508992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573349.206816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495508992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573349.206822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573349.206824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573349.206826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495508992:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573349.206831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495509120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573349.206846:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573349.206853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff93a258da1880 x1875453495509120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573349.206867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495509120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573349.206872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573349.206874:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573349.206877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495509120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573350.166351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573350.166394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788573351.190338:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573351.190363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573352.214363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573352.214436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573353.238342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573353.238351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573354.262327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573354.262439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573354.326347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.326356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573354.326360:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.326368:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1570 00000100:00080000:3.0:1788573354.326377:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.326383:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573354.326416:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495509248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573354.326445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.326449:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573354.326451:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.326456:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1570 00000100:00080000:3.0:1788573354.326460:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.326464:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573354.326474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573354.326494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495509248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573354.326507:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495509376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573354.326528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.326531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573354.326533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.326536:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1570 00000100:00080000:3.0:1788573354.326540:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.326543:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573354.326547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573354.326552:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573354.326558:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495509504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573354.326569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.326571:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573354.326573:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.326576:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1570 00000100:00080000:3.0:1788573354.326579:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.326582:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573354.326585:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573354.326593:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495509632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573354.326603:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff93a25064df80 x1875453495509248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573354.326618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495509248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573354.326626:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573354.326630:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573354.326632:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495509248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573354.326660:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495509376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573354.326677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573354.326685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff93a25064ca80 x1875453495509376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573354.326694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495509376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573354.326699:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573354.326701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573354.326704:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495509376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573354.326708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495509504:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573354.326719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573354.326725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff93a25064c700 x1875453495509504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573354.326732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495509504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573354.326748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573354.326751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573354.326753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495509504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573354.326757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495509632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573354.326770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573354.326776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff93a25064d500 x1875453495509632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573354.326783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495509632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573354.326788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573354.326790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573354.326792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495509632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573355.286372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573355.286416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573356.310369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573356.310426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573357.334372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573357.334451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573358.358347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573358.358395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573359.382362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573359.382457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573359.446356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.446365:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573359.446368:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.446377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1575 00000100:00080000:2.0:1788573359.446385:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.446391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573359.446437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495509760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.446461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.446464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573359.446466:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.446469:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1575 00000100:00080000:2.0:1788573359.446472:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.446475:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573359.446483:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573359.446492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495509888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.446498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.446501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573359.446502:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.446504:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1575 00000100:00080000:2.0:1788573359.446506:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.446508:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573359.446512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573359.446518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495510016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.446526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.446530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573359.446532:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573359.446532:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495509760:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573359.446542:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1575 00000100:00080000:2.0:1788573359.446544:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.446547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573359.446551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573359.446559:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495510144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573359.446591:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573359.446654:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff93a24940aa00 x1875453495509760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573359.446671:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495509760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573359.446680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573359.446684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573359.446688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495509760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573359.446694:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495509888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573359.446721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573359.446731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff93a249408700 x1875453495509888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573359.446740:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495509888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573359.446745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573359.446748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573359.446750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495509888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573359.446769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510016:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573359.446782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573359.446788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff93a249409880 x1875453495510016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573359.446803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495510016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573359.446809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573359.446811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573359.446813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510016:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573359.446817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495510144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573359.446839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573359.446846:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff93a249409500 x1875453495510144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573359.446854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495510144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573359.446862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573359.446865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573359.446868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495510144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573360.406314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573360.406358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573361.430281:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573361.430342:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573362.454312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573362.454367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573363.478304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573363.478363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573364.502322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573364.502375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573364.566333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.566342:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573364.566345:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.566353:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1581 00000100:00080000:0.0:1788573364.566362:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.566368:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573364.566400:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495510272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573364.566423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.566426:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573364.566427:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.566430:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1581 00000100:00080000:0.0:1788573364.566432:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.566435:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573364.566442:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573364.566450:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495510400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573364.566458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.566460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573364.566476:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.566482:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1581 00000100:00100000:3.0:1788573364.566482:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495510272:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573364.566486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.566492:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573364.566496:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573364.566505:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495510528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573364.566512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.566515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573364.566517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.566519:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1581 00000100:00080000:0.0:1788573364.566522:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.566524:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573364.566527:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573364.566534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495510656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573364.566547:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573364.566584:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff93a2460d3480 x1875453495510272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573364.566606:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495510272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573364.566613:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573364.566617:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573364.566620:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495510272:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573364.566625:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573364.566642:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573364.566651:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff93a2460d0380 x1875453495510400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573364.566659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495510400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573364.566664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573364.566666:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573364.566668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573364.566686:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510528:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573364.566698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573364.566704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff93a2460d2680 x1875453495510528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573364.566711:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495510528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573364.566725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573364.566727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573364.566730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573364.566734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495510656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573364.566746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573364.566753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff93a2460d1500 x1875453495510656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573364.566761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495510656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573364.566766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573364.566767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573364.566769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495510656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573365.526389:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573365.526468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573366.550349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573366.550421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573367.574369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573367.574419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573368.598408:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573368.598434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573369.622353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573369.622432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573369.686403:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.686418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573369.686425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.686438:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1586 00000100:00080000:0.0:1788573369.686454:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.686463:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573369.686520:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495510784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573369.686569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.686572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573369.686573:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.686577:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1586 00000100:00080000:0.0:1788573369.686581:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.686585:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573369.686596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573369.686611:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495510912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573369.686623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.686630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573369.686631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495510784:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573369.686633:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.686643:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1586 00000100:00080000:0.0:1788573369.686648:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.686653:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573369.686660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573369.686683:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495511040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573369.686696:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.686700:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573369.686702:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.686707:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1586 00000100:00080000:0.0:1788573369.686717:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.686720:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573369.686725:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573369.686737:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495511168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573369.686737:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573369.686802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff93a2460d0a80 x1875453495510784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573369.686824:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495510784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573369.686844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573369.686851:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573369.686856:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495510784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573369.686887:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573369.686915:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573369.686927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff93a2460d0000 x1875453495510912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573369.686936:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495510912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573369.686943:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573369.686945:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573369.686947:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495510912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573369.686952:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511040:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573369.686966:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573369.686973:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff93a2460d3b80 x1875453495511040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573369.686981:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495511040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573369.686986:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573369.686988:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573369.686990:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511040:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573369.687002:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495511168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573369.687016:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573369.687024:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff93a2460d0700 x1875453495511168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573369.687032:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495511168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573369.687038:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573369.687039:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573369.687042:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495511168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573370.646335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573370.646384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573371.670379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573371.670453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573372.694343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573372.694364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573373.718342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573373.718409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573374.742348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573374.742355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573374.806337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573374.806346:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573374.806350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573374.806359:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1591 00000100:00080000:0.0:1788573374.806370:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573374.806376:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573374.806411:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495511296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573374.806434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573374.806437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573374.806439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573374.806447:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1591 00000100:00080000:0.0:1788573374.806450:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573374.806453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573374.806461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573374.806470:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495511424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573374.806478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573374.806496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573374.806499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573374.806499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495511296:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573374.806506:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1591 00000100:00080000:0.0:1788573374.806511:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573374.806514:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573374.806518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573374.806527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495511552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573374.806535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573374.806538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573374.806540:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573374.806542:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1591 00000100:00080000:0.0:1788573374.806546:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573374.806549:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573374.806551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573374.806557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495511680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573374.806574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573374.806624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff93a2460d0700 x1875453495511296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573374.806655:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495511296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573374.806667:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573374.806672:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573374.806676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495511296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573374.806693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573374.806731:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573374.806747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff93a2460d3b80 x1875453495511424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573374.806763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495511424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573374.806772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573374.806776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573374.806780:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573374.806805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511552:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573374.806830:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573374.806843:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff93a2460d0000 x1875453495511552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573374.806858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495511552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573374.806868:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573374.806871:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573374.806885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511552:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573374.806893:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495511680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573374.806915:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573374.806928:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff93a2460d0a80 x1875453495511680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573374.806951:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495511680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573374.806960:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573374.806964:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573374.806968:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495511680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573375.766341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573375.766391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573376.790359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573376.790386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573377.814360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573377.814360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573378.838385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573378.838453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573379.862348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573379.862352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573379.926357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573379.926365:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573379.926369:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573379.926377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1596 00000100:00080000:1.0:1788573379.926386:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573379.926408:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573379.926437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495511808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573379.926462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573379.926465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573379.926466:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573379.926469:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1596 00000100:00080000:1.0:1788573379.926472:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573379.926474:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573379.926482:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573379.926491:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495511936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573379.926499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573379.926501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573379.926503:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573379.926507:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1596 00000100:00100000:3.0:1788573379.926511:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495511808:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573379.926519:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573379.926523:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573379.926526:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573379.926534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495512064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573379.926540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573379.926543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573379.926549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573379.926553:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1596 00000100:00080000:1.0:1788573379.926557:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573379.926559:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573379.926563:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573379.926571:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495512192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573379.926579:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573379.926624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff93a258da3100 x1875453495511808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573379.926639:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495511808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573379.926650:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573379.926654:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573379.926659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495511808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573379.926685:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573379.926711:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573379.926725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff93a258da2a00 x1875453495511936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573379.926747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495511936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573379.926757:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573379.926761:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573379.926763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495511936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573379.926768:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512064:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573379.926785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573379.926793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff93a258da3480 x1875453495512064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573379.926801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495512064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573379.926806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573379.926808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573379.926810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573379.926814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495512192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573379.926824:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573379.926833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff93a258da0380 x1875453495512192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573379.926844:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495512192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573379.926865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573379.926869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573379.926873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495512192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573380.886368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573380.886406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573381.910381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573381.910448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573382.934336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573382.934383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573383.958341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573383.958398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573384.982379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573384.982451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573385.046434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573385.046448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573385.046455:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573385.046470:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1601 00000100:00080000:2.0:1788573385.046483:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573385.046492:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573385.046544:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495512320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573385.046594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573385.046599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573385.046601:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573385.046606:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1601 00000100:00080000:2.0:1788573385.046610:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573385.046615:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573385.046627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573385.046645:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495512448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573385.046652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495512320:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573385.046670:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573385.046675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573385.046677:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573385.046683:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1601 00000100:00080000:2.0:1788573385.046688:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573385.046693:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573385.046719:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573385.046732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573385.046736:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495512576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573385.046750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573385.046754:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573385.046757:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573385.046762:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1601 00000100:00080000:2.0:1788573385.046766:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573385.046771:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573385.046776:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573385.046789:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495512704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573385.046796:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff93a24940b480 x1875453495512320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573385.046822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495512320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573385.046849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573385.046854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573385.046858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495512320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573385.046888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573385.046909:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573385.046920:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff93a24940bb80 x1875453495512448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573385.046928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495512448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573385.046934:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573385.046937:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573385.046939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573385.046942:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512576:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573385.046955:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573385.046961:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff93a24940b100 x1875453495512576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573385.046969:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495512576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573385.046974:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573385.046976:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573385.046978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512576:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573385.046987:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495512704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573385.047001:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573385.047008:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff93a249409500 x1875453495512704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573385.047015:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495512704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573385.047020:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573385.047022:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573385.047024:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495512704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573386.006387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573386.006451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573387.030361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573387.030401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573388.054313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573388.054367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573389.078330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573389.078396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573390.102350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573390.102388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573390.166326:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.166335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573390.166338:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.166346:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1606 00000100:00080000:3.0:1788573390.166355:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.166361:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573390.166394:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495512832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573390.166421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.166424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573390.166426:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.166429:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1606 00000100:00080000:3.0:1788573390.166431:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.166433:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573390.166442:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573390.166450:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495512960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573390.166470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.166473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573390.166474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.166478:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1606 00000100:00080000:3.0:1788573390.166481:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.166484:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573390.166488:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573390.166496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495513088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573390.166502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495512832:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573390.166506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.166510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573390.166512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.166515:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1606 00000100:00080000:3.0:1788573390.166517:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.166519:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573390.166522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573390.166530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495513216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573390.166562:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573390.166608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff93a25064f100 x1875453495512832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573390.166628:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495512832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573390.166640:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573390.166645:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573390.166650:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495512832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573390.166666:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573390.166685:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573390.166695:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff93a25064c380 x1875453495512960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573390.166704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495512960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573390.166710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573390.166712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573390.166714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495512960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573390.166733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513088:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573390.166745:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573390.166751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff93a25064e300 x1875453495513088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573390.166758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495513088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573390.166763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573390.166765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573390.166767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513088:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573390.166770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495513216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573390.166780:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573390.166786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff93a25064ce00 x1875453495513216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573390.166799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495513216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573390.166805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573390.166807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573390.166809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495513216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573391.126333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573391.126391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573392.150312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573392.150343:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573393.174320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573393.174375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573394.198386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573394.198527:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573395.222577:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573395.222753:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573395.286377:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573395.286389:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573395.286394:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573395.286405:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1611 00000100:00080000:3.0:1788573395.286439:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573395.286447:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573395.286486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495513344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573395.286514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573395.286517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573395.286519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573395.286522:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1611 00000100:00080000:3.0:1788573395.286525:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573395.286527:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573395.286537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573395.286546:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495513472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573395.286554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573395.286557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573395.286559:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573395.286563:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1611 00000100:00080000:3.0:1788573395.286565:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573395.286567:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573395.286571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573395.286577:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495513600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573395.286586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573395.286595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573395.286597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573395.286602:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495513344:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573395.286602:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1611 00000100:00080000:3.0:1788573395.286612:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573395.286618:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573395.286622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573395.286633:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495513728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573395.286712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573395.286742:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573395.286762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495513728:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573396.246826:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573396.246899:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573397.270622:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573397.270734:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573398.294436:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573398.294891:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573398.358508:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788573398.358561:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573398.358575:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573398.358615:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573398.358626:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573398.358635:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788573398.358676:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff93a25064d180 x1875453495513600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.358703:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495513600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.358715:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.358720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573398.358733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573398.358742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff93a25064c000 x1875453495513728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.358757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495513728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.358766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.358770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573398.358774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495513728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573398.358781:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff93a25064f480 x1875453495513344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.358794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495513344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.358801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.358804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573398.358808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495513344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573398.358814:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff93a25064fb80 x1875453495513472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.358826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495513472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.358835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.358838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573398.358849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513472:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573399.318796:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573399.319110:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788573400.342346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573400.342429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573400.406380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573400.406390:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573400.406395:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573400.406405:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1616 00000100:00080000:2.0:1788573400.406415:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573400.406423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573400.406456:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495513856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573400.406481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573400.406485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573400.406487:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573400.406491:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1616 00000100:00080000:2.0:1788573400.406495:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573400.406499:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573400.406509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573400.406523:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495513984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573400.406534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573400.406538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495513856:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573400.406554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573400.406556:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573400.406562:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1616 00000100:00080000:2.0:1788573400.406567:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573400.406572:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573400.406579:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573400.406592:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495514112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573400.406604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573400.406610:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573400.406612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573400.406615:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573400.406621:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1616 00000100:00080000:2.0:1788573400.406627:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573400.406632:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573400.406637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573400.406649:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495514240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573400.406662:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff93a249409880 x1875453495513856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.406694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495513856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.406707:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.406718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573400.406723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495513856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573400.406750:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573400.406777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573400.406793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff93a249408700 x1875453495513984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.406807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495513984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.406816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.406820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573400.406823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495513984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573400.406830:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495514112:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573400.406854:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573400.406878:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff93a24940aa00 x1875453495514112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.406891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495514112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.406899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.406902:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573400.406906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495514112:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573400.406913:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495514240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573400.406936:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573400.406949:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff93a249408a80 x1875453495514240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.406960:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495514240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.406969:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.406972:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573400.406975:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495514240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573401.366381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573401.366419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573402.390346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573402.390408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573403.414327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573403.414401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573404.438377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573404.438453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573405.462381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573405.462442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573405.526362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573405.526371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573405.526392:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573405.526403:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1621 00000100:00080000:2.0:1788573405.526414:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573405.526421:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573405.526456:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495514368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573405.526485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573405.526489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573405.526491:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573405.526496:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1621 00000100:00080000:2.0:1788573405.526501:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573405.526506:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573405.526518:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573405.526550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495514496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573405.526560:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495514368:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573405.526570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573405.526575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573405.526577:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573405.526583:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1621 00000100:00080000:2.0:1788573405.526587:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573405.526591:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573405.526599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573405.526615:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495514624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573405.526641:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573405.526645:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573405.526647:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573405.526652:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1621 00000100:00080000:2.0:1788573405.526656:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573405.526664:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573405.526665:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573405.526672:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573405.526691:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495514752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573405.526754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff93a249409180 x1875453495514368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.526778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495514368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.526792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.526799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573405.526804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495514368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573405.526836:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495514496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573405.526866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573405.526882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff93a249408000 x1875453495514496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.526895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495514496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.526902:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.526905:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573405.526907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495514496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573405.526912:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495514624:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573405.526930:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573405.526938:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff93a24940ad80 x1875453495514624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.526946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495514624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.526961:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.526964:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573405.526966:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495514624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573405.526970:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495514752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573405.526986:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573405.526997:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff93a249409c00 x1875453495514752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.527019:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495514752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.527031:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.527035:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573405.527038:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495514752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573406.486390:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573406.486440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573407.510330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573407.510448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573408.534354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573408.534462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573409.558293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573409.558431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573410.582331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573410.582381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573410.646361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.646370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573410.646374:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.646382:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1627 00000100:00080000:0.0:1788573410.646391:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.646397:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573410.646458:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495514880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.646490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.646493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573410.646495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.646498:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1627 00000100:00080000:0.0:1788573410.646501:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.646504:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573410.646512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573410.646521:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495515008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.646529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.646532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573410.646533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.646536:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1627 00000100:00080000:0.0:1788573410.646538:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.646540:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573410.646543:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573410.646548:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495514880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573410.646553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495515136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.646565:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.646567:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573410.646569:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.646577:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1627 00000100:00080000:0.0:1788573410.646580:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.646583:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573410.646587:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573410.646596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495515264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573410.646620:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573410.646665:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff93a2460d1500 x1875453495514880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573410.646680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495514880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573410.646687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573410.646690:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573410.646693:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495514880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573410.646712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573410.646733:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573410.646757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff93a2460d2680 x1875453495515008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573410.646771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495515008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573410.646779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573410.646781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573410.646784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573410.646788:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515136:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573410.646805:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573410.646813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff93a2460d0380 x1875453495515136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573410.646820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495515136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573410.646825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573410.646827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573410.646829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515136:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573410.646833:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495515264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573410.646842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573410.646848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff93a2460d3480 x1875453495515264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573410.646854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495515264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573410.646865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573410.646867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573410.646869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495515264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573411.606286:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573411.606372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573412.630334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573412.630391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573413.654279:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573413.654351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573414.678350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573414.678428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573415.702372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573415.702444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573415.766323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.766336:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573415.766340:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.766351:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1632 00000100:00080000:3.0:1788573415.766363:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.766371:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573415.766421:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495515392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.766453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.766457:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573415.766460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.766466:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1632 00000100:00080000:3.0:1788573415.766471:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.766477:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573415.766489:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495515392:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788573415.766490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573415.766520:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495515520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.766547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.766550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573415.766551:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.766556:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1632 00000100:00080000:3.0:1788573415.766559:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.766563:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573415.766568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573415.766578:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495515648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.766586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.766589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1788573415.766593:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573415.766593:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.766599:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1632 00000100:00080000:3.0:1788573415.766602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.766605:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573415.766609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573415.766618:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495515776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573415.766650:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff93a25064d880 x1875453495515392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.766673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495515392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.766686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.766691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573415.766696:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495515392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573415.766726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573415.766746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573415.766756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff93a25064e680 x1875453495515520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.766779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495515520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.766785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.766788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573415.766790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573415.766794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515648:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573415.766809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573415.766817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff93a25064ed80 x1875453495515648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.766825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495515648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.766830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.766832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573415.766835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495515648:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573415.766841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495515776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573415.766861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573415.766873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff93a25064dc00 x1875453495515776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.766886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495515776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.766910:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.766912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573415.766915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495515776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573416.726343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573416.726415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573417.750331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573417.750384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573418.774345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573418.774414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573419.798369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573419.798428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573420.822322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573420.822397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573420.886336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573420.886351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573420.886355:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573420.886363:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1637 00000100:00080000:1.0:1788573420.886372:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573420.886377:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573420.886409:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495515904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573420.886432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573420.886435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573420.886436:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573420.886439:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1637 00000100:00080000:1.0:1788573420.886441:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573420.886444:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573420.886451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573420.886474:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495516032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573420.886482:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573420.886486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788573420.886489:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495515904:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573420.886489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573420.886494:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1637 00000100:00080000:1.0:1788573420.886499:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573420.886501:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573420.886504:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573420.886513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495516160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573420.886520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573420.886523:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573420.886524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573420.886527:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1637 00000100:00080000:1.0:1788573420.886547:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573420.886549:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573420.886552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573420.886556:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573420.886564:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495516288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573420.886610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff93a258da1180 x1875453495515904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573420.886632:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495515904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573420.886644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573420.886648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573420.886652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495515904:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573420.886676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573420.886699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573420.886723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff93a258da0000 x1875453495516032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573420.886738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495516032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573420.886752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573420.886754:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573420.886757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573420.886761:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516160:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573420.886778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573420.886789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff93a258da2300 x1875453495516160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573420.886801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495516160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573420.886810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573420.886813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573420.886817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573420.886824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495516288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573420.886839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573420.886847:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff93a258da0380 x1875453495516288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573420.886855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495516288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573420.886861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573420.886863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573420.886872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495516288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573421.846340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573421.846386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573422.870368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573422.870449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573423.894345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573423.894401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573424.918325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573424.918422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573425.942326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573425.942382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573426.006386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.006398:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573426.006403:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.006414:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1642 00000100:00080000:0.0:1788573426.006427:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.006435:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573426.006481:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495516416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.006513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.006517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573426.006519:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.006525:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1642 00000100:00080000:0.0:1788573426.006528:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.006534:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573426.006547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573426.006562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495516544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.006579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573426.006580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495516416:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573426.006591:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573426.006595:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.006601:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1642 00000100:00080000:0.0:1788573426.006608:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.006634:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573426.006642:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573426.006654:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495516672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.006664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573426.006665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573426.006674:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573426.006678:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.006684:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1642 00000100:00080000:0.0:1788573426.006721:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.006728:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573426.006735:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573426.006757:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495516800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573426.006758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff93a2460d2a00 x1875453495516416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573426.006820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495516416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573426.006833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573426.006838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573426.006843:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495516416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573426.006870:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573426.006898:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573426.006909:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff93a2460d1180 x1875453495516544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573426.006918:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495516544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573426.006924:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573426.006927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573426.006929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573426.006948:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516672:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573426.006965:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573426.006972:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff93a2460d2300 x1875453495516672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573426.006980:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495516672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573426.006985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573426.006987:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573426.006989:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495516672:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573426.006993:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495516800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573426.007004:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573426.007010:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff93a2460d1f80 x1875453495516800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573426.007017:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495516800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573426.007022:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573426.007024:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573426.007026:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495516800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573426.966328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573426.966381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573427.990369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573427.990432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573429.014324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573429.014417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573430.038310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573430.038378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573431.062323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573431.062376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573431.126320:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573431.126329:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573431.126332:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573431.126340:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1647 00000100:00080000:2.0:1788573431.126350:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573431.126356:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573431.126392:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495516928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573431.126426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573431.126428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573431.126430:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573431.126433:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1647 00000100:00080000:2.0:1788573431.126436:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573431.126438:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573431.126446:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573431.126457:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495517056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573431.126465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573431.126468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573431.126469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573431.126474:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1647 00000100:00100000:3.0:1788573431.126476:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495516928:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573431.126477:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573431.126483:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573431.126487:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573431.126495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495517184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573431.126502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573431.126504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573431.126505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573431.126508:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1647 00000100:00080000:2.0:1788573431.126515:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573431.126518:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573431.126522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573431.126530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495517312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573431.126548:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573431.126595:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff93a24940b800 x1875453495516928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573431.126611:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495516928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573431.126619:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573431.126623:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573431.126626:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495516928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573431.126646:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573431.126663:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573431.126672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff93a249408e00 x1875453495517056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573431.126679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495517056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573431.126685:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573431.126687:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573431.126690:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573431.126693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517184:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573431.126704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573431.126710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff93a24940a680 x1875453495517184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573431.126725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495517184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573431.126731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573431.126734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573431.126737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573431.126744:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495517312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573431.126765:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573431.126777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff93a249409c00 x1875453495517312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573431.126815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495517312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573431.126821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573431.126823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573431.126826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495517312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573432.086321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573432.086381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573433.110354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573433.110405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573434.134328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573434.134410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573435.158357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573435.158424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573436.182310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573436.182419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573436.246328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573436.246336:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573436.246339:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573436.246347:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1652 00000100:00080000:0.0:1788573436.246356:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573436.246362:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573436.246410:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495517440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573436.246432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573436.246435:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573436.246436:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573436.246439:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1652 00000100:00080000:0.0:1788573436.246442:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573436.246445:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573436.246453:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573436.246468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495517568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573436.246477:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573436.246481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573436.246486:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573436.246486:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495517440:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573436.246497:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1652 00000100:00080000:0.0:1788573436.246501:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573436.246506:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573436.246512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573436.246527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495517696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573436.246537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573436.246541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788573436.246545:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573436.246553:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573436.246564:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1652 00000100:00080000:0.0:1788573436.246568:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573436.246571:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573436.246577:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573436.246586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495517824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573436.246593:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff93a2460d1f80 x1875453495517440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573436.246618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495517440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573436.246630:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573436.246634:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573436.246639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495517440:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573436.246665:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573436.246687:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573436.246697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff93a2460d2300 x1875453495517568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573436.246706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495517568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573436.246711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573436.246714:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573436.246716:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573436.246720:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517696:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573436.246732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573436.246738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff93a2460d1180 x1875453495517696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573436.246746:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495517696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573436.246760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573436.246763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573436.246765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495517696:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573436.246769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495517824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573436.246782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573436.246790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff93a2460d2a00 x1875453495517824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573436.246797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495517824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573436.246802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573436.246804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573436.246806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495517824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573437.206330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573437.206384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573438.230348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573438.230406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573439.254337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573439.254439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573440.278341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573440.278414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573441.302321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573441.302410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573441.366281:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573441.366289:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573441.366293:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573441.366301:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1657 00000100:00080000:0.0:1788573441.366309:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573441.366315:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573441.366344:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495517952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573441.366365:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573441.366368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573441.366369:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573441.366372:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1657 00000100:00080000:0.0:1788573441.366374:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573441.366377:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573441.366397:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495517952:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573441.366398:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573441.366412:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495518080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573441.366421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573441.366423:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573441.366425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573441.366428:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1657 00000100:00080000:0.0:1788573441.366431:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573441.366434:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573441.366438:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573441.366445:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495518208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573441.366452:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573441.366456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788573441.366457:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573441.366458:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573441.366464:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1657 00000100:00080000:0.0:1788573441.366467:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573441.366470:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573441.366474:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573441.366482:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495518336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573441.366502:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff93a2460d3480 x1875453495517952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573441.366517:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495517952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573441.366530:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573441.366534:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573441.366537:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495517952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573441.366556:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573441.366574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573441.366583:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff93a2460d0380 x1875453495518080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573441.366591:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495518080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573441.366596:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573441.366598:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573441.366600:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573441.366604:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518208:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573441.366614:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573441.366620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff93a2460d2680 x1875453495518208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573441.366627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495518208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573441.366633:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573441.366635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573441.366636:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518208:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573441.366645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495518336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573441.366659:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573441.366666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff93a2460d1500 x1875453495518336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573441.366673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495518336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573441.366678:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573441.366680:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573441.366681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495518336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573442.326325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573442.326372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573443.350386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573443.350441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573444.374333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573444.374413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573445.398333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573445.398445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573446.422333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573446.422428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573446.486333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.486345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573446.486350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.486362:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1662 00000100:00080000:0.0:1788573446.486374:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.486383:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573446.486433:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495518464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573446.486458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.486461:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573446.486462:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.486466:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1662 00000100:00080000:0.0:1788573446.486469:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.486471:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573446.486482:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573446.486492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495518592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573446.486499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495518464:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573446.486503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.486508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573446.486509:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.486513:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1662 00000100:00080000:0.0:1788573446.486515:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.486518:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573446.486521:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573446.486555:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495518720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573446.486563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.486567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573446.486568:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.486572:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1662 00000100:00080000:0.0:1788573446.486575:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.486578:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573446.486582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573446.486587:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573446.486593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495518848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573446.486642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff93a2460d0a80 x1875453495518464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573446.486664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495518464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573446.486674:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573446.486678:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573446.486682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495518464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573446.486712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573446.486732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573446.486742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff93a2460d0000 x1875453495518592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573446.486751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495518592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573446.486757:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573446.486759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573446.486762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573446.486765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518720:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573446.486777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573446.486791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff93a2460d3b80 x1875453495518720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573446.486799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495518720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573446.486804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573446.486807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573446.486809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495518720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573446.486812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495518848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573446.486823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573446.486830:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff93a2460d0700 x1875453495518848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573446.486849:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495518848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573446.486855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573446.486857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573446.486860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495518848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573447.446324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573447.446455:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573448.470320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573448.470402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573449.494332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573449.494420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573450.518306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573450.518375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573451.542363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573451.542640:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573451.606347:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573451.606355:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573451.606359:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573451.606367:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1668 00000100:00080000:3.0:1788573451.606375:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573451.606381:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573451.606413:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495518976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573451.606435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573451.606438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573451.606440:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573451.606442:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1668 00000100:00080000:3.0:1788573451.606445:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573451.606462:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573451.606470:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495518976:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788573451.606471:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573451.606485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495519104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573451.606494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573451.606497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573451.606498:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573451.606501:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1668 00000100:00080000:3.0:1788573451.606504:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573451.606506:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573451.606509:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573451.606517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495519232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573451.606525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573451.606528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573451.606529:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573451.606531:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1668 00000100:00080000:3.0:1788573451.606533:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573451.606536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573451.606539:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573451.606546:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495519360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573451.606554:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573451.606612:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff93a25064ea00 x1875453495518976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573451.606633:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495518976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573451.606644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573451.606648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573451.606652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495518976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573451.606677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573451.606703:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573451.606728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff93a25064dc00 x1875453495519104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573451.606738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495519104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573451.606745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573451.606748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573451.606752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573451.606758:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519232:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573451.606791:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573451.606810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff93a25064ed80 x1875453495519232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573451.606822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495519232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573451.606829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573451.606832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573451.606835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519232:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573451.606841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495519360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573451.606860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573451.606870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff93a25064e680 x1875453495519360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573451.606883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495519360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573451.606895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573451.606898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573451.606900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495519360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573452.566308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573452.566380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573453.590362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573453.590392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573454.614374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573454.614463:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573455.638325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573455.638368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573456.662337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573456.662403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573456.726383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573456.726393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573456.726397:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573456.726422:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1673 00000100:00080000:1.0:1788573456.726430:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573456.726436:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573456.726470:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495519488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573456.726495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573456.726498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573456.726499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573456.726502:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1673 00000100:00080000:1.0:1788573456.726505:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573456.726507:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573456.726515:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573456.726523:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495519616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573456.726530:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573456.726533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573456.726534:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573456.726537:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1673 00000100:00080000:1.0:1788573456.726539:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573456.726541:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573456.726544:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573456.726550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495519744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573456.726560:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495519488:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573456.726574:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573456.726577:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573456.726579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573456.726582:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1673 00000100:00080000:1.0:1788573456.726585:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573456.726588:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573456.726591:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573456.726599:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495519872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573456.726648:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573456.726693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff93a258da3480 x1875453495519488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.726714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495519488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.726725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.726730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573456.726734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495519488:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573456.726742:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573456.726766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573456.726780:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff93a258da2a00 x1875453495519616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.726794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495519616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.726803:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.726807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573456.726810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573456.726845:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519744:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573456.726869:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573456.726881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff93a258da3100 x1875453495519744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.726894:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495519744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.726903:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.726906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573456.726910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495519744:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573456.726917:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495519872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573456.726936:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573456.726947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff93a258da1880 x1875453495519872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.726959:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495519872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.726967:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.726971:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573456.726974:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495519872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573457.686326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573457.686371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573458.710388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573458.710469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573459.734338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573459.734393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573460.758500:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573460.758538:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573461.782689:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573461.782824:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:1.0:1788573461.846389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573461.846405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573461.846410:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573461.846425:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1678 00000100:00080000:1.0:1788573461.846439:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573461.846449:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573461.846500:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495520000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573461.846535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573461.846539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573461.846541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573461.846546:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1678 00000100:00080000:1.0:1788573461.846550:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573461.846555:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573461.846582:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495520000:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788573461.846584:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573461.846602:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495520128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573461.846611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573461.846614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573461.846616:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573461.846619:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1678 00000100:00080000:1.0:1788573461.846623:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573461.846626:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573461.846631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573461.846639:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495520256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573461.846647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573461.846650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573461.846651:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573461.846654:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1678 00000100:00080000:1.0:1788573461.846657:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573461.846659:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573461.846662:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573461.846669:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495520384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573461.846705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573461.846746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573461.846773:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495520384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573462.806373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573462.806436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573463.830837:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573463.830872:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573464.854699:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573464.854913:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:3.0:1788573464.918501:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788573464.918546:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573464.918563:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788573464.918612:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788573464.918623:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788573464.918632:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788573464.918672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573464] req@ffff93a258da1c00 x1875453495520128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573464.918700:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495520128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573464.918713:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573464.918719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573464.918724:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573464.918733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573464] req@ffff93a258da0700 x1875453495520256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573464.918746:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495520256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573464.918755:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573464.918759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573464.918763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573464.918770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573464] req@ffff93a258da0a80 x1875453495520384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573464.918783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495520384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573464.918815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573464.918820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573464.918824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495520384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573464.918831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573464] req@ffff93a258da2680 x1875453495520000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573464.918847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495520000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573464.918855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573464.918859:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573464.918862:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495520000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573465.878421:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573465.879070:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573466.902384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573466.902433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573466.966341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573466.966353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573466.966375:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573466.966385:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1683 00000100:00080000:1.0:1788573466.966394:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573466.966402:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573466.966439:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495520512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573466.966465:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573466.966468:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573466.966471:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573466.966476:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1683 00000100:00080000:1.0:1788573466.966480:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573466.966485:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573466.966496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573466.966510:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495520640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573466.966532:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495520512:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573466.966532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573466.966547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573466.966551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573466.966558:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1683 00000100:00080000:1.0:1788573466.966562:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573466.966567:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573466.966574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573466.966592:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495520768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573466.966617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573466.966621:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573466.966624:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573466.966629:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1683 00000100:00080000:1.0:1788573466.966635:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573466.966640:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788573466.966651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788573466.966662:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573466.966685:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495520896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573466.966729:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff93a258da3800 x1875453495520512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573466.966757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495520512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573466.966770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573466.966775:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573466.966779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495520512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573466.966815:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573466.966864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573466.966879:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff93a258da0e00 x1875453495520640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573466.966893:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495520640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573466.966910:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573466.966914:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573466.966918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573466.966926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520768:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573466.966952:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573466.966965:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff93a258da1f80 x1875453495520768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573466.966978:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495520768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573466.966987:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573466.966991:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573466.966994:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495520768:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573466.967001:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495520896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573466.967022:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573466.967033:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff93a258da2680 x1875453495520896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573466.967082:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495520896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573466.967098:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573466.967102:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573466.967106:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495520896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573467.926341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573467.926432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573468.950357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573468.950433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573469.974310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573469.974409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573470.998335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573470.998448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573472.022322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573472.022441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573472.086384:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573472.086394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573472.086397:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573472.086406:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1688 00000100:00080000:0.0:1788573472.086414:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573472.086420:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573472.086452:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495521024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573472.086478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573472.086481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573472.086483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573472.086486:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1688 00000100:00080000:0.0:1788573472.086488:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573472.086490:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573472.086498:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573472.086507:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495521152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573472.086518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495521024:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573472.086539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573472.086542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573472.086543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573472.086548:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1688 00000100:00080000:0.0:1788573472.086551:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573472.086554:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573472.086558:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573472.086568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495521280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573472.086576:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573472.086577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573472.086581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573472.086583:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573472.086587:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1688 00000100:00080000:0.0:1788573472.086597:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573472.086601:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573472.086606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573472.086618:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495521408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573472.086626:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff93a2460d0700 x1875453495521024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573472.086653:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495521024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573472.086673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573472.086677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573472.086680:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495521024:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573472.086709:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573472.086728:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573472.086742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff93a2460d3b80 x1875453495521152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573472.086755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495521152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573472.086764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573472.086768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573472.086771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573472.086777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521280:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573472.086804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573472.086814:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff93a2460d0000 x1875453495521280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573472.086822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495521280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573472.086828:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573472.086830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573472.086832:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573472.086844:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495521408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573472.086870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573472.086879:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff93a2460d0a80 x1875453495521408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573472.086887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495521408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573472.086892:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573472.086894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573472.086896:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495521408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573473.046345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573473.046401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573474.070328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573474.070459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573475.094338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573475.094402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573476.118373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573476.118436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573477.142345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573477.142400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573477.206371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573477.206384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573477.206389:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573477.206400:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1693 00000100:00080000:0.0:1788573477.206427:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573477.206435:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573477.206467:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495521536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573477.206491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573477.206494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573477.206496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573477.206500:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1693 00000100:00080000:0.0:1788573477.206504:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573477.206509:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573477.206518:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573477.206532:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495521536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573477.206533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495521664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573477.206548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573477.206551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573477.206553:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573477.206558:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1693 00000100:00080000:0.0:1788573477.206561:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573477.206565:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573477.206570:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573477.206580:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495521792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573477.206592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573477.206604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788573477.206604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573477.206608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573477.206616:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1693 00000100:00080000:0.0:1788573477.206622:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573477.206627:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573477.206634:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573477.206647:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495521920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573477.206656:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff93a2460d1500 x1875453495521536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573477.206679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495521536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573477.206689:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573477.206694:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573477.206698:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495521536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573477.206728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573477.206752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573477.206765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff93a2460d2680 x1875453495521664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573477.206778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495521664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573477.206787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573477.206790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573477.206794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573477.206801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521792:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573477.206820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573477.206832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff93a2460d0380 x1875453495521792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573477.206845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495521792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573477.206853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573477.206857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573477.206860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495521792:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573477.206865:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495521920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573477.206878:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573477.206885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff93a2460d3480 x1875453495521920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573477.206899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495521920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573477.206905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573477.206907:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573477.206909:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495521920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573478.166350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573478.166443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573479.190335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573479.190390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573480.214381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573480.214427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573481.238321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573481.238413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573482.262356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573482.262425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573482.326361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.326373:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573482.326378:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.326388:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1698 00000100:00080000:0.0:1788573482.326400:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.326408:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573482.326450:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495522048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.326484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.326489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573482.326491:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.326496:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1698 00000100:00080000:0.0:1788573482.326501:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.326506:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573482.326518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573482.326534:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495522048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573482.326544:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495522176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.326574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.326578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573482.326580:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.326586:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1698 00000100:00080000:0.0:1788573482.326591:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573482.326596:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573482.326601:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573482.326609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573482.326624:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495522304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.326636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573482.326637:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff93a2460d2a00 x1875453495522048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573482.326647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573482.326652:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.326658:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1698 00000100:00100000:3.0:1788573482.326663:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495522048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573482.326664:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.326673:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:3.0:1788573482.326678:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788573482.326680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788573482.326684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573482.326689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495522048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573482.326698:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495522432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573482.326711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573482.326729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573482.326739:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff93a2460d1180 x1875453495522176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573482.326769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495522176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573482.326776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573482.326778:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573482.326781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573482.326785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522304:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573482.326800:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573482.326808:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff93a2460d2300 x1875453495522304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573482.326815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495522304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573482.326821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573482.326823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573482.326825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522304:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573482.326834:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495522432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573482.326851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573482.326862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff93a2460d1f80 x1875453495522432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573482.326875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495522432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573482.326895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573482.326898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573482.326900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495522432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573483.286345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573483.286450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573484.310376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573484.310465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573485.334325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573485.334398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573486.358357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573486.358478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573487.382299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573487.382396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573487.446334:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.446344:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573487.446347:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.446356:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1703 00000100:00080000:0.0:1788573487.446382:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.446390:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573487.446425:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495522560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.446449:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.446452:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573487.446453:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.446460:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1703 00000100:00080000:0.0:1788573487.446463:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.446467:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573487.446475:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573487.446485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495522688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.446496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573487.446501:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495522560:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573487.446502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573487.446507:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.446525:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1703 00000100:00080000:0.0:1788573487.446533:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.446537:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573487.446542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573487.446553:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495522816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.446561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.446569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573487.446571:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.446574:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1703 00000100:00080000:0.0:1788573487.446577:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.446580:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573487.446585:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573487.446585:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573487.446597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495522944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573487.446630:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff93a2460d1f80 x1875453495522560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.446645:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495522560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.446653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.446656:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573487.446659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495522560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573487.446677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573487.446694:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573487.446702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff93a2460d2300 x1875453495522688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.446710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495522688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.446722:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.446725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573487.446727:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573487.446731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522816:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573487.446746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573487.446752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff93a2460d1180 x1875453495522816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.446760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495522816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.446765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.446767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573487.446768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495522816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573487.446772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495522944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573487.446782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573487.446787:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff93a2460d2a00 x1875453495522944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573487.446794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495522944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573487.446799:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573487.446801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573487.446807:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495522944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573488.406343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573488.406401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573489.430316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573489.430325:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573490.454334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573490.454383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573491.478335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573491.478481:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573492.502308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573492.502411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573492.566356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573492.566365:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573492.566368:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573492.566376:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1709 00000100:00080000:2.0:1788573492.566385:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573492.566391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573492.566435:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495523072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573492.566458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573492.566460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573492.566462:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573492.566465:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1709 00000100:00080000:2.0:1788573492.566468:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573492.566470:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573492.566477:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573492.566488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495523200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573492.566496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573492.566500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573492.566501:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495523072:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573492.566502:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573492.566510:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1709 00000100:00080000:2.0:1788573492.566512:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573492.566516:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573492.566519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573492.566525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495523328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573492.566532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573492.566534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573492.566536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573492.566544:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1709 00000100:00080000:2.0:1788573492.566547:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573492.566550:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573492.566554:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573492.566562:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495523456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573492.566574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573492.566610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff93a24940ad80 x1875453495523072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573492.566624:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495523072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573492.566632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573492.566635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573492.566638:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495523072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573492.566655:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573492.566670:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573492.566678:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff93a249408000 x1875453495523200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573492.566686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495523200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573492.566691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573492.566693:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573492.566701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573492.566706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523328:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573492.566720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573492.566727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff93a249409180 x1875453495523328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573492.566735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495523328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573492.566740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573492.566741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573492.566743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523328:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573492.566746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495523456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573492.566756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573492.566762:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff93a249408a80 x1875453495523456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573492.566770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495523456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573492.566774:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573492.566776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573492.566778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495523456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573493.526309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573493.526384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573494.550302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573494.550362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573495.574334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573495.574398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573496.598308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573496.598361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573497.622341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573497.622403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573497.686312:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573497.686321:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573497.686325:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573497.686333:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1714 00000100:00080000:2.0:1788573497.686341:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573497.686348:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573497.686380:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495523584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573497.686405:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573497.686409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573497.686410:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573497.686415:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1714 00000100:00080000:2.0:1788573497.686419:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573497.686423:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573497.686450:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573497.686458:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495523584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573497.686468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495523712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573497.686482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573497.686486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573497.686488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573497.686494:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1714 00000100:00080000:2.0:1788573497.686508:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573497.686514:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573497.686520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573497.686533:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573497.686535:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495523840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573497.686549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573497.686554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573497.686556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573497.686560:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1714 00000100:00080000:2.0:1788573497.686565:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573497.686569:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573497.686575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573497.686589:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495523968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573497.686592:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff93a24940aa00 x1875453495523584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573497.686626:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495523584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573497.686637:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573497.686642:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573497.686646:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495523584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573497.686667:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573497.686689:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573497.686701:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff93a249408700 x1875453495523712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573497.686713:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495523712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573497.686721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573497.686725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573497.686728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573497.686734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523840:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573497.686765:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573497.686776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff93a249409880 x1875453495523840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573497.686787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495523840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573497.686795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573497.686799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573497.686802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495523840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573497.686808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495523968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573497.686825:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573497.686835:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff93a249409500 x1875453495523968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573497.686846:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495523968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573497.686853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573497.686857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573497.686860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495523968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573498.646329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573498.646382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573499.670365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573499.670405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573500.694327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573500.694384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573501.718348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573501.718419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573502.742362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573502.742438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573502.806306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573502.806317:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573502.806322:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573502.806332:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1719 00000100:00080000:2.0:1788573502.806341:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573502.806347:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573502.806380:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495524096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573502.806406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573502.806408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573502.806410:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573502.806413:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1719 00000100:00080000:2.0:1788573502.806415:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573502.806417:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573502.806425:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573502.806434:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495524224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573502.806441:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573502.806444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573502.806445:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573502.806448:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1719 00000100:00080000:2.0:1788573502.806450:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573502.806454:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495524096:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573502.806471:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573502.806476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573502.806486:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495524352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573502.806495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573502.806497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573502.806499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573502.806502:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1719 00000100:00080000:2.0:1788573502.806512:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573502.806514:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573502.806518:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573502.806522:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573502.806529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495524480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573502.806569:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff93a24940b100 x1875453495524096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573502.806585:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495524096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573502.806593:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573502.806596:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573502.806600:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495524096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573502.806609:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573502.806630:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573502.806638:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff93a24940bb80 x1875453495524224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573502.806646:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495524224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573502.806652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573502.806653:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573502.806655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573502.806674:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524352:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573502.806686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573502.806693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff93a24940b480 x1875453495524352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573502.806712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495524352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573502.806718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573502.806720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573502.806722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524352:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573502.806726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495524480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573502.806739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573502.806746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff93a249409500 x1875453495524480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573502.806753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495524480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573502.806758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573502.806760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573502.806761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495524480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573503.766358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573503.766446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573504.790335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573504.790440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573505.814332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573505.814353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573506.838341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573506.838424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573507.862324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573507.862380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573507.926329:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.926337:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573507.926341:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.926349:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1724 00000100:00080000:2.0:1788573507.926358:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.926364:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573507.926395:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495524608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573507.926430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.926433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573507.926435:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.926440:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1724 00000100:00080000:2.0:1788573507.926444:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.926448:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573507.926458:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573507.926472:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495524736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573507.926480:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495524608:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573507.926483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.926486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573507.926488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.926491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1724 00000100:00080000:2.0:1788573507.926494:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.926497:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573507.926503:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573507.926511:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495524864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573507.926520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.926523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573507.926524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.926528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1724 00000100:00080000:2.0:1788573507.926538:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573507.926540:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573507.926542:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573507.926547:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573507.926556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495524992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573507.926592:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff93a249409880 x1875453495524608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573507.926623:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495524608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573507.926631:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573507.926634:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573507.926638:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495524608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573507.926658:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573507.926677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573507.926686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff93a249408700 x1875453495524736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573507.926694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495524736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573507.926700:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573507.926712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573507.926714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573507.926718:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524864:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573507.926732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573507.926739:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff93a24940aa00 x1875453495524864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573507.926747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495524864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573507.926761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573507.926763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573507.926765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495524864:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573507.926769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495524992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573507.926783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573507.926789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff93a249408a80 x1875453495524992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573507.926796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495524992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573507.926801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573507.926803:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573507.926805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495524992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573508.886311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573508.886399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573509.910326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573509.910375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573510.934350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573510.934513:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573511.958331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573511.958389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573512.982336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573512.982396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573513.046321:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573513.046329:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573513.046333:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573513.046341:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1729 00000100:00080000:2.0:1788573513.046350:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573513.046356:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573513.046387:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495525120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573513.046408:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573513.046411:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573513.046412:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573513.046415:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1729 00000100:00080000:2.0:1788573513.046417:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573513.046420:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573513.046427:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573513.046450:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495525120:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573513.046453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495525248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573513.046463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573513.046465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573513.046467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573513.046470:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1729 00000100:00080000:2.0:1788573513.046473:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573513.046475:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573513.046479:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573513.046487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495525376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573513.046494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573513.046496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573513.046498:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573513.046500:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1729 00000100:00080000:2.0:1788573513.046503:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573513.046505:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573513.046508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573513.046514:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495525504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573513.046519:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573513.046563:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff93a249409180 x1875453495525120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573513.046577:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495525120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573513.046592:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573513.046595:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573513.046599:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495525120:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573513.046618:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573513.046637:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573513.046646:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff93a249408000 x1875453495525248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573513.046654:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495525248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573513.046660:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573513.046662:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573513.046665:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573513.046668:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525376:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573513.046679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573513.046685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff93a24940ad80 x1875453495525376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573513.046692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495525376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573513.046697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573513.046699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573513.046701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525376:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573513.046710:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495525504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573513.046723:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573513.046730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff93a249409c00 x1875453495525504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573513.046737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495525504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573513.046743:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573513.046744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573513.046752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495525504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573514.006296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573514.006356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573515.030337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573515.030389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573516.054333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573516.054382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573517.078400:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573517.078529:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573518.102362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573518.102427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573518.166376:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573518.166388:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573518.166392:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573518.166403:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1734 00000100:00080000:0.0:1788573518.166415:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573518.166422:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573518.166468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495525632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573518.166497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573518.166501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573518.166502:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573518.166505:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1734 00000100:00080000:0.0:1788573518.166508:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573518.166511:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573518.166521:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573518.166530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495525760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573518.166546:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495525632:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573518.166563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573518.166566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573518.166567:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573518.166572:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1734 00000100:00080000:0.0:1788573518.166575:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573518.166578:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573518.166582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573518.166593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495525888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573518.166600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573518.166603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573518.166604:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573518.166606:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1734 00000100:00080000:0.0:1788573518.166616:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573518.166619:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573518.166622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573518.166624:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573518.166632:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495526016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573518.166684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff93a2460d3480 x1875453495525632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573518.166704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495525632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573518.166715:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573518.166718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573518.166722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495525632:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573518.166756:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573518.166782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573518.166798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff93a2460d0380 x1875453495525760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573518.166808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495525760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573518.166814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573518.166817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573518.166819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573518.166823:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525888:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573518.166840:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573518.166847:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff93a2460d2680 x1875453495525888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573518.166855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495525888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573518.166860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573518.166868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573518.166870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495525888:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573518.166874:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495526016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573518.166886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573518.166893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff93a2460d1500 x1875453495526016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573518.166907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495526016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573518.166913:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573518.166915:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573518.166917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495526016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573519.126308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573519.126351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573520.150302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573520.150364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573521.174355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573521.174416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573522.198304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573522.198352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573523.222348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573523.222381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573523.286365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573523.286375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573523.286379:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573523.286388:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1739 00000100:00080000:2.0:1788573523.286397:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573523.286403:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573523.286435:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495526144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573523.286456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573523.286459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573523.286460:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573523.286463:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1739 00000100:00080000:2.0:1788573523.286465:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573523.286468:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573523.286475:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573523.286484:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495526272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573523.286492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573523.286495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573523.286504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495526144:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573523.286511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573523.286517:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1739 00000100:00080000:2.0:1788573523.286520:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573523.286523:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573523.286527:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573523.286536:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495526400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573523.286544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573523.286548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573523.286551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573523.286555:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1739 00000100:00080000:2.0:1788573523.286559:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573523.286563:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573523.286568:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573523.286574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573523.286584:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495526528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573523.286614:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff93a24940a680 x1875453495526144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573523.286628:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495526144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573523.286637:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573523.286640:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573523.286643:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495526144:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573523.286662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573523.286679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573523.286687:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff93a249408e00 x1875453495526272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573523.286706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495526272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573523.286715:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573523.286719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573523.286722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573523.286729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526400:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573523.286752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573523.286763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff93a24940b800 x1875453495526400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573523.286772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495526400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573523.286777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573523.286780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573523.286782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526400:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573523.286786:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495526528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573523.286799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573523.286805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff93a249409c00 x1875453495526528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573523.286812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495526528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573523.286829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573523.286831:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573523.286833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495526528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573524.246320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573524.246368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573525.270332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573525.270392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573526.294294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573526.294360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573527.318400:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573527.318426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573528.342632:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573528.342693:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788573528.406358:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573528.406366:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573528.406370:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573528.406378:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1744 00000100:00080000:0.0:1788573528.406402:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573528.406408:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573528.406438:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495526656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573528.406461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573528.406464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573528.406465:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573528.406468:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1744 00000100:00080000:0.0:1788573528.406471:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573528.406473:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573528.406481:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573528.406490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495526784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573528.406498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573528.406502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573528.406503:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573528.406503:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495526656:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573528.406511:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1744 00000100:00080000:0.0:1788573528.406514:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573528.406516:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573528.406520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573528.406529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495526912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573528.406537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573528.406543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573528.406545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573528.406548:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1744 00000100:00080000:0.0:1788573528.406559:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573528.406562:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573528.406566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573528.406575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495527040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573528.406607:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573528.406635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526912:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573528.406652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495527040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573529.366496:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573529.366794:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573530.390423:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573530.391278:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573531.414782:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573531.414869:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788573531.478673:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788573531.478740:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573531.478763:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573531.478822:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573531.478835:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573531.478848:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573531.478890:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff93a2460d0000 x1875453495526784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573531.478926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495526784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573531.478943:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573531.478949:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573531.478954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573531.478964:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff93a2460d3b80 x1875453495526912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573531.479010:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495526912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573531.479018:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573531.479021:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573531.479025:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495526912:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573531.479033:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff93a2460d0700 x1875453495527040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573531.479045:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495527040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573531.479052:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573531.479056:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573531.479059:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495527040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573531.479067:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff93a2460d0a80 x1875453495526656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573531.479081:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495526656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573531.479088:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573531.479093:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573531.479096:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495526656:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573532.438671:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573532.438815:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573533.462319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573533.462342:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573533.526353:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573533.526365:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573533.526371:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573533.526382:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1749 00000100:00080000:0.0:1788573533.526395:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573533.526402:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573533.526447:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495527168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573533.526476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573533.526480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573533.526483:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573533.526488:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1749 00000100:00080000:0.0:1788573533.526492:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573533.526497:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573533.526510:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573533.526525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495527296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573533.526536:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495527168:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573533.526564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573533.526568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573533.526571:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573533.526579:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1749 00000100:00080000:0.0:1788573533.526583:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573533.526587:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573533.526593:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573533.526604:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495527424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573533.526613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573533.526616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573533.526618:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573533.526622:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1749 00000100:00080000:0.0:1788573533.526626:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573533.526633:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788573533.526633:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788573533.526643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573533.526656:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495527552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573533.526725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff93a2460d0a80 x1875453495527168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573533.526749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495527168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573533.526763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573533.526767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573533.526771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495527168:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573533.526802:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573533.526833:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573533.526849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff93a2460d0700 x1875453495527296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573533.526862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495527296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573533.526870:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573533.526873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573533.526877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573533.526883:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527424:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573533.526905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573533.526916:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff93a2460d3b80 x1875453495527424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573533.526947:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495527424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573533.526955:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573533.526959:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573533.526962:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527424:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573533.526968:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495527552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573533.526992:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573533.527005:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff93a2460d0000 x1875453495527552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573533.527017:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495527552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573533.527026:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573533.527029:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573533.527032:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495527552:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788573534.486322:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573534.486331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573535.510312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573535.510327:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573536.534289:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573536.534315:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573537.558314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573537.558414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573538.582344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573538.582373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573538.646309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573538.646321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573538.646325:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573538.646335:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1755 00000100:00080000:0.0:1788573538.646358:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573538.646365:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573538.646394:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495527680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573538.646416:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573538.646418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573538.646420:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573538.646423:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1755 00000100:00080000:0.0:1788573538.646426:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573538.646429:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573538.646438:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573538.646442:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495527680:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573538.646449:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495527808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573538.646458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573538.646460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573538.646461:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573538.646464:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1755 00000100:00080000:0.0:1788573538.646466:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573538.646468:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573538.646472:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573538.646480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495527936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573538.646488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573538.646495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573538.646497:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573538.646502:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573538.646505:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1755 00000100:00080000:0.0:1788573538.646510:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573538.646514:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573538.646519:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573538.646528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495528064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573538.646542:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff93a2460d1500 x1875453495527680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573538.646565:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495527680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573538.646572:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573538.646575:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573538.646578:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495527680:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573538.646597:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573538.646615:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573538.646624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff93a2460d2680 x1875453495527808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573538.646631:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495527808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573538.646637:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573538.646639:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573538.646641:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573538.646645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527936:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573538.646656:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573538.646661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff93a2460d0380 x1875453495527936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573538.646669:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495527936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573538.646682:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573538.646685:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573538.646687:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495527936:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573538.646691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495528064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573538.646704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573538.646711:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff93a2460d3480 x1875453495528064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573538.646718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495528064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573538.646723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573538.646724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573538.646726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495528064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573539.606368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573539.606398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573540.630329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573540.630351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573541.654309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573541.654338:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573542.678360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573542.678382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573543.702319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573543.702337:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573543.766385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573543.766399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573543.766404:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573543.766415:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1760 00000100:00080000:0.0:1788573543.766428:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573543.766436:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573543.766480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495528192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573543.766509:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573543.766512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573543.766513:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573543.766517:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1760 00000100:00080000:0.0:1788573543.766520:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573543.766522:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573543.766533:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573543.766549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495528320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573543.766580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495528192:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573543.766585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573543.766597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573543.766600:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573543.766606:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1760 00000100:00080000:0.0:1788573543.766609:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573543.766612:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573543.766617:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573543.766633:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495528448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573543.766644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573543.766649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573543.766651:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573543.766656:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1760 00000100:00080000:0.0:1788573543.766662:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573543.766670:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788573543.766672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788573543.766683:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573543.766697:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495528576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573543.766731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff93a2460d2a00 x1875453495528192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573543.766753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495528192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573543.766778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573543.766783:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573543.766788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495528192:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573543.766812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573543.766837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573543.766850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff93a2460d1180 x1875453495528320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573543.766863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495528320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573543.766873:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573543.766876:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573543.766880:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573543.766887:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528448:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573543.766906:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573543.766918:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff93a2460d2300 x1875453495528448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573543.766931:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495528448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573543.766940:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573543.766945:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573543.766948:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528448:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573543.766962:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495528576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573543.766983:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573543.766994:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff93a2460d1f80 x1875453495528576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573543.767008:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495528576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573543.767017:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573543.767021:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573543.767025:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495528576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573544.726339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573544.726365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573545.750328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573545.750379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573546.774362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573546.774402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573547.798290:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573547.798315:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573548.822338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573548.822373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573548.886400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573548.886412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573548.886417:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573548.886428:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1765 00000100:00080000:0.0:1788573548.886455:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573548.886464:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573548.886501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495528704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573548.886528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573548.886532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573548.886534:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573548.886539:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1765 00000100:00080000:0.0:1788573548.886543:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573548.886549:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573548.886560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573548.886564:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495528704:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573548.886577:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495528832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573548.886590:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573548.886594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573548.886596:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573548.886601:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1765 00000100:00080000:0.0:1788573548.886606:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573548.886610:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573548.886616:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573548.886621:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573548.886633:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495528960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573548.886646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573548.886657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff93a2460d1f80 x1875453495528704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573548.886658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573548.886662:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573548.886672:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1765 00000100:00080000:0.0:1788573548.886679:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573548.886681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495528704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573548.886687:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:3.0:1788573548.886693:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788573548.886697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788573548.886699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573548.886703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495528704:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573548.886729:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495529088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573548.886738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573548.886772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573548.886785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff93a2460d2300 x1875453495528832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573548.886798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495528832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573548.886807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573548.886811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573548.886815:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573548.886822:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528960:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573548.886844:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573548.886855:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff93a2460d1180 x1875453495528960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573548.886868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495528960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573548.886877:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573548.886881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573548.886885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495528960:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573548.886900:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495529088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573548.886922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573548.886934:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff93a2460d2a00 x1875453495529088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573548.886957:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495529088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573548.886963:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573548.886965:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573548.886967:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495529088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573549.846347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573549.846372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573550.870323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573550.870346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573551.894385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573551.894402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573552.918353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573552.918388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573553.942357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573553.942402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573554.006337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573554.006345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573554.006349:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573554.006358:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1770 00000100:00080000:3.0:1788573554.006366:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573554.006373:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573554.006404:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495529216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573554.006428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573554.006431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573554.006433:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573554.006436:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1770 00000100:00080000:3.0:1788573554.006438:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573554.006457:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573554.006468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573554.006470:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495529216:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573554.006486:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495529344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573554.006500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573554.006505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573554.006508:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573554.006513:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1770 00000100:00080000:3.0:1788573554.006519:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573554.006524:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573554.006531:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573554.006534:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573554.006551:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495529472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573554.006566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573554.006570:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573554.006572:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573554.006578:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1770 00000100:00100000:0.0:1788573554.006579:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff93a25064d880 x1875453495529216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573554.006584:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573554.006594:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573554.006602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573554.006605:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495529216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573554.006618:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573554.006620:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495529600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573554.006625:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573554.006633:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495529216:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573554.006662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573554.006686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573554.006698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff93a25064fb80 x1875453495529344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573554.006709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495529344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573554.006725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573554.006728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573554.006732:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573554.006738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529472:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573554.006758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573554.006768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff93a25064f480 x1875453495529472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573554.006779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495529472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573554.006787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573554.006791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573554.006794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529472:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573554.006800:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495529600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573554.006820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573554.006830:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff93a25064c000 x1875453495529600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573554.006842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495529600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573554.006850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573554.006853:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573554.006862:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495529600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573554.966352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573554.966371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573555.990333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573555.990354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573557.014328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573557.014359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573558.038393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573558.038503:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573559.062331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573559.062364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573559.126362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573559.126378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573559.126384:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573559.126400:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1775 00000100:00080000:3.0:1788573559.126413:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573559.126421:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573559.126467:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495529728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573559.126494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573559.126497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573559.126498:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573559.126502:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1775 00000100:00080000:3.0:1788573559.126507:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573559.126511:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573559.126522:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573559.126536:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495529856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573559.126538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495529728:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573559.126559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573559.126563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573559.126564:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573559.126568:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1775 00000100:00080000:3.0:1788573559.126591:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573559.126596:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573559.126600:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573559.126610:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495529984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573559.126619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573559.126622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573559.126623:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573559.126630:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1775 00000800:00000100:0.0:1788573559.126632:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573559.126635:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573559.126639:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573559.126643:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573559.126654:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495530112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573559.126688:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff93a25064d180 x1875453495529728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573559.126712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495529728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573559.126725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573559.126730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573559.126741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495529728:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573559.126770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573559.126796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573559.126807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff93a25064ce00 x1875453495529856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573559.126816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495529856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573559.126825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573559.126829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573559.126832:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573559.126839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529984:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573559.126862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573559.126886:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff93a25064e300 x1875453495529984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573559.126895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495529984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573559.126900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573559.126903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573559.126905:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495529984:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573559.126912:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495530112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573559.126934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573559.126947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff93a25064c380 x1875453495530112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573559.126959:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495530112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573559.126968:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573559.126971:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573559.126974:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495530112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573560.086334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573560.086363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573561.110334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573561.110354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573562.134348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573562.134378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573563.158315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573563.158332:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573564.182352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573564.182389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573564.246343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573564.246355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573564.246359:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573564.246370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1780 00000100:00080000:3.0:1788573564.246381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573564.246388:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573564.246440:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495530240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573564.246468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573564.246472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573564.246474:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573564.246478:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1780 00000100:00080000:3.0:1788573564.246482:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573564.246486:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573564.246495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573564.246507:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495530368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573564.246519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573564.246523:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573564.246525:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573564.246530:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1780 00000100:00080000:3.0:1788573564.246533:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573564.246536:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573564.246540:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573564.246551:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495530496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573564.246568:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495530240:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573564.246570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573564.246582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573564.246588:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573564.246598:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1780 00000100:00080000:3.0:1788573564.246603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573564.246608:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573564.246614:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573564.246627:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495530624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573564.246678:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573564.246734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff93a25064f100 x1875453495530240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573564.246752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495530240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573564.246777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573564.246783:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573564.246788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495530240:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573564.246798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495530368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573564.246836:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573564.246848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff93a25064d500 x1875453495530368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573564.246858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495530368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573564.246864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573564.246867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573564.246869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495530368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573564.246896:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495530496:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573564.246912:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573564.246919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff93a25064c700 x1875453495530496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573564.246935:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495530496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573564.246941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573564.246944:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573564.246946:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495530496:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573564.246956:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495530624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573564.246972:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573564.246980:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff93a25064ca80 x1875453495530624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573564.246988:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495530624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573564.246993:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573564.246995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573564.246997:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495530624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573565.206309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573565.206343:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573566.230341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573566.230369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573567.254336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573567.254352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573568.278352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573568.278387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573569.302360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573569.302395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573569.366366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573569.366396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573569.366402:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573569.366414:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1785 00000100:00080000:3.0:1788573569.366427:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573569.366433:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573569.366478:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495530752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573569.366517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573569.366522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573569.366524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573569.366530:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1785 00000100:00080000:3.0:1788573569.366534:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573569.366539:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573569.366574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573569.366576:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495530752:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573569.366596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495530880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573569.366610:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573569.366614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573569.366616:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573569.366620:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1785 00000100:00080000:3.0:1788573569.366625:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573569.366629:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573569.366635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573569.366648:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495531008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573569.366659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573569.366663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573569.366666:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573569.366668:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573569.366673:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1785 00000100:00080000:3.0:1788573569.366679:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573569.366684:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573569.366690:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573569.366703:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495531136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573569.366727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff93a25064df80 x1875453495530752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573569.366751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495530752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573569.366764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573569.366768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573569.366772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495530752:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573569.366804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495530880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573569.366828:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573569.366857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff93a25064ca80 x1875453495530880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573569.366880:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495530880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573569.366890:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573569.366893:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573569.366897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495530880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573569.366904:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531008:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573569.366928:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573569.366941:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff93a25064c700 x1875453495531008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573569.366953:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495531008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573569.366961:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573569.366965:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573569.366968:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531008:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573569.366974:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495531136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573569.366993:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573569.367004:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff93a25064d500 x1875453495531136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573569.367016:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495531136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573569.367037:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573569.367041:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573569.367045:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495531136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573570.326339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573570.326368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573571.350366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573571.350391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573572.374324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573572.374392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573573.398343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573573.398372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573574.422298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573574.422332:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573574.486321:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.486334:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573574.486339:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.486349:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1790 00000100:00080000:0.0:1788573574.486358:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.486365:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573574.486397:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495531264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.486420:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.486423:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.486425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.486429:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1790 00000100:00080000:0.0:1788573574.486431:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.486435:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573574.486443:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573574.486448:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495531264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573574.486454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495531392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.486463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.486465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.486480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.486484:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1790 00000100:00080000:0.0:1788573574.486488:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.486491:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573574.486496:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573574.486508:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495531520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573574.486508:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573574.486519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.486521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.486523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.486527:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1790 00000100:00080000:0.0:1788573574.486529:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.486532:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573574.486535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573574.486543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495531648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573574.486548:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff93a2460d3480 x1875453495531264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573574.486566:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495531264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573574.486573:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573574.486577:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573574.486581:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495531264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573574.486599:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573574.486613:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573574.486621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff93a2460d0380 x1875453495531392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573574.486629:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495531392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573574.486634:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573574.486637:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573574.486639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573574.486642:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531520:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573574.486665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573574.486672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff93a2460d2680 x1875453495531520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573574.486679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495531520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573574.486685:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573574.486686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573574.486688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531520:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573574.486692:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495531648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573574.486701:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573574.486707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff93a2460d1500 x1875453495531648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573574.486714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495531648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573574.486722:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573574.486725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573574.486729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495531648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573575.446333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573575.446376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573576.470331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573576.470391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573577.494364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573577.494459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573578.518360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573578.518439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573579.542340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573579.542405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573579.606388:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573579.606397:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573579.606400:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573579.606409:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1796 00000100:00080000:2.0:1788573579.606417:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573579.606423:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573579.606457:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495531776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573579.606479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573579.606483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573579.606485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573579.606489:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1796 00000100:00080000:2.0:1788573579.606493:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573579.606497:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573579.606507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573579.606522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495531904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573579.606528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495531776:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573579.606532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573579.606535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573579.606537:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573579.606540:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1796 00000100:00080000:2.0:1788573579.606544:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573579.606566:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573579.606571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573579.606581:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495532032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573579.606587:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573579.606591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573579.606594:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573579.606596:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573579.606599:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1796 00000100:00080000:2.0:1788573579.606603:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573579.606607:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573579.606611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573579.606620:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495532160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573579.606647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff93a24940ad80 x1875453495531776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573579.606668:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495531776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573579.606679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573579.606685:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573579.606689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495531776:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573579.606713:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573579.606740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573579.606751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff93a249408000 x1875453495531904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573579.606760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495531904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573579.606777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573579.606780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573579.606783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495531904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573579.606790:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532032:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573579.606806:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573579.606813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff93a249409180 x1875453495532032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573579.606828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495532032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573579.606834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573579.606836:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573579.606838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532032:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573579.606842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495532160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573579.606854:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573579.606861:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff93a249408a80 x1875453495532160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573579.606869:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495532160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573579.606874:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573579.606876:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573579.606878:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495532160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573580.566359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573580.566437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573581.590372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573581.590431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573582.614336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573582.614421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573583.638372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573583.638450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573584.662341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573584.662413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573584.726432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573584.726444:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573584.726449:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573584.726460:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1801 00000100:00080000:2.0:1788573584.726473:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573584.726481:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573584.726534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495532288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573584.726580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573584.726583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573584.726585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573584.726588:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1801 00000100:00080000:2.0:1788573584.726592:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573584.726595:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573584.726605:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573584.726616:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495532416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573584.726625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573584.726628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573584.726629:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573584.726632:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1801 00000100:00080000:2.0:1788573584.726634:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573584.726637:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573584.726640:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573584.726649:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495532544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573584.726651:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495532288:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573584.726661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573584.726665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573584.726666:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573584.726669:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1801 00000100:00080000:2.0:1788573584.726676:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573584.726679:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573584.726684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573584.726693:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495532672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573584.726746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573584.726804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff93a24940aa00 x1875453495532288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573584.726836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495532288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573584.726848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573584.726854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573584.726858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495532288:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573584.726867:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573584.726896:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573584.726913:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff93a249408700 x1875453495532416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573584.726926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495532416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573584.726935:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573584.726939:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573584.726942:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573584.726974:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532544:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573584.726999:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573584.727012:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff93a249409880 x1875453495532544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573584.727034:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495532544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573584.727056:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573584.727060:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573584.727064:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532544:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573584.727071:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495532672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573584.727095:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573584.727107:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff93a249409500 x1875453495532672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573584.727119:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495532672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573584.727128:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573584.727131:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573584.727134:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495532672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573585.686342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573585.686413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573586.710322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573586.710374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573587.734336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573587.734410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573588.758330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573588.758409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573589.782374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573589.782434:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788573589.846358:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.846370:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573589.846375:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.846385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1806 00000100:00080000:0.0:1788573589.846396:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.846403:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573589.846442:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495532800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573589.846495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.846507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573589.846510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.846515:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1806 00000100:00080000:0.0:1788573589.846521:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.846526:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573589.846537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573589.846541:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495532800:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573589.846556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495532928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573589.846569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.846574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573589.846576:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.846581:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1806 00000100:00080000:0.0:1788573589.846586:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.846590:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573589.846596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573589.846609:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573589.846611:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495533056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573589.846626:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.846630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573589.846632:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.846637:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1806 00000100:00080000:0.0:1788573589.846648:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.846653:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573589.846659:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573589.846661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff93a2460d0000 x1875453495532800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573589.846678:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495533184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573589.846682:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495532800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573589.846701:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573589.846705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573589.846708:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495532800:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573589.846735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573589.846755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573589.846764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff93a2460d3b80 x1875453495532928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573589.846773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495532928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573589.846779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573589.846781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573589.846784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495532928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573589.846787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533056:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573589.846798:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573589.846804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff93a2460d0700 x1875453495533056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573589.846812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495533056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573589.846819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573589.846822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573589.846825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533056:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573589.846838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495533184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573589.846861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573589.846874:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff93a2460d0a80 x1875453495533184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573589.846887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495533184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573589.846896:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573589.846900:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573589.846903:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495533184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573590.806340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573590.806395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573591.830316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573591.830397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573592.854354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573592.854405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573593.878352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573593.878449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573594.902732:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573594.903291:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573594.966324:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573594.966332:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573594.966336:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573594.966344:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1811 00000100:00080000:3.0:1788573594.966351:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573594.966357:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573594.966386:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495533312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573594.966408:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573594.966411:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573594.966425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573594.966429:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1811 00000100:00080000:3.0:1788573594.966432:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573594.966436:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573594.966445:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573594.966449:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495533312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573594.966457:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495533440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573594.966465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573594.966467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573594.966469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573594.966471:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1811 00000100:00080000:3.0:1788573594.966473:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573594.966475:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573594.966478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573594.966487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495533568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573594.966495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573594.966498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573594.966499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573594.966502:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1811 00000100:00080000:3.0:1788573594.966504:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573594.966506:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573594.966509:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573594.966521:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495533696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573594.966563:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573594.966592:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573594.966607:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495533696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573595.926448:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573595.926615:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573596.950711:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573596.950944:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573597.974623:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573597.974808:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:1.0:1788573598.038668:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788573598.038719:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573598.038737:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573598.038781:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573598.038794:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573598.038807:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788573598.038862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573598] req@ffff93a25064c380 x1875453495533440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573598.038908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495533440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573598.038925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573598.038933:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573598.038938:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573598.038951:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573598] req@ffff93a25064e300 x1875453495533568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573598.038967:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495533568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573598.039004:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573598.039010:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573598.039014:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573598.039025:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573598] req@ffff93a25064ce00 x1875453495533696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573598.039040:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495533696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573598.039050:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573598.039054:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573598.039058:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495533696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573598.039066:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573598] req@ffff93a25064f100 x1875453495533312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573598.039120:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495533312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573598.039130:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573598.039134:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573598.039137:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495533312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573598.998441:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573598.999392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573600.022338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573600.022404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573600.086318:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573600.086327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573600.086331:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573600.086340:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1816 00000100:00080000:2.0:1788573600.086348:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573600.086354:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573600.086383:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495533824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573600.086420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573600.086423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573600.086425:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573600.086428:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1816 00000100:00080000:2.0:1788573600.086431:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573600.086433:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573600.086441:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573600.086451:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495533952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573600.086459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573600.086461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573600.086462:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573600.086467:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1816 00000100:00100000:0.0:1788573600.086469:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495533824:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573600.086470:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573600.086474:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573600.086480:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573600.086488:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495534080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573600.086494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573600.086497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573600.086499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573600.086501:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1816 00000100:00080000:2.0:1788573600.086508:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573600.086511:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573600.086515:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573600.086524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495534208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573600.086546:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573600.086589:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff93a24940b480 x1875453495533824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573600.086625:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495533824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573600.086636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573600.086640:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573600.086644:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495533824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573600.086676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573600.086707:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573600.086722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff93a24940bb80 x1875453495533952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573600.086736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495533952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573600.086745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573600.086748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573600.086751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495533952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573600.086773:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534080:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573600.086798:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573600.086811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff93a24940b100 x1875453495534080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573600.086823:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495534080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573600.086845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573600.086849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573600.086852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534080:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573600.086859:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495534208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573600.086893:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573600.086907:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff93a249409500 x1875453495534208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573600.086920:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495534208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573600.086928:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573600.086931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573600.086934:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495534208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573601.046353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573601.046449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573602.070306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573602.070371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573603.094339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573603.094405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573604.118293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573604.118357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573605.142321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573605.142395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573605.206309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573605.206319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573605.206324:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573605.206334:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1821 00000100:00080000:2.0:1788573605.206344:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573605.206351:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573605.206384:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495534336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573605.206406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573605.206408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573605.206409:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573605.206413:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1821 00000100:00080000:2.0:1788573605.206415:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573605.206418:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573605.206426:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573605.206446:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495534464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573605.206454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573605.206456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573605.206458:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573605.206463:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1821 00000100:00100000:0.0:1788573605.206465:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495534336:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573605.206466:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573605.206472:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573605.206478:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573605.206486:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495534592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573605.206493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573605.206495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573605.206497:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573605.206499:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1821 00000100:00080000:2.0:1788573605.206501:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573605.206503:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573605.206506:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573605.206516:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495534720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573605.206534:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573605.206572:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff93a249409880 x1875453495534336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573605.206592:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495534336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573605.206603:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573605.206608:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573605.206613:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495534336:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573605.206646:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573605.206672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573605.206685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff93a249408700 x1875453495534464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573605.206709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495534464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573605.206723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573605.206727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573605.206731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573605.206738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534592:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573605.206760:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573605.206772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff93a24940aa00 x1875453495534592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573605.206785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495534592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573605.206794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573605.206798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573605.206801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573605.206818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495534720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573605.206840:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573605.206851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff93a249408a80 x1875453495534720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573605.206881:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495534720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573605.206891:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573605.206895:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573605.206899:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495534720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573606.166360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573606.166454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573607.190361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573607.190577:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573608.214348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573608.214439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573609.238324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573609.238395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573610.262368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573610.262454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573610.326308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573610.326317:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573610.326320:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573610.326344:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1826 00000100:00080000:2.0:1788573610.326353:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573610.326359:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573610.326396:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495534848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573610.326428:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573610.326431:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573610.326433:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573610.326436:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1826 00000100:00080000:2.0:1788573610.326439:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573610.326442:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573610.326451:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573610.326462:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495534976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573610.326469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573610.326472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573610.326473:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573610.326476:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1826 00000100:00080000:2.0:1788573610.326478:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573610.326480:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573610.326483:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573610.326488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495534848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573610.326493:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495535104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573610.326514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573610.326516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573610.326517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573610.326520:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1826 00000100:00080000:2.0:1788573610.326524:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573610.326527:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573610.326530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573610.326539:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495535232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573610.326557:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573610.326591:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff93a249409180 x1875453495534848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573610.326605:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495534848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573610.326613:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573610.326616:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573610.326619:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495534848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573610.326625:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573610.326641:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573610.326649:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff93a249408000 x1875453495534976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573610.326665:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495534976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573610.326672:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573610.326674:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573610.326676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495534976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573610.326695:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495535104:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573610.326710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573610.326717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff93a24940ad80 x1875453495535104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573610.326725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495535104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573610.326730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573610.326731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573610.326733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495535104:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573610.326737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495535232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573610.326761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573610.326768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff93a249409c00 x1875453495535232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573610.326776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495535232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573610.326791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573610.326793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573610.326796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495535232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573611.286335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573611.286389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573612.310379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573612.310456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573613.334313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573613.334392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573614.358326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573614.358388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573615.382393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573615.382431:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788573615.446361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.446370:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573615.446374:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.446383:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1831 00000100:00080000:0.0:1788573615.446392:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.446398:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573615.446430:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495535360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573615.446455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.446458:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573615.446460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.446473:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1831 00000100:00080000:0.0:1788573615.446475:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.446478:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573615.446487:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573615.446496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495535488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573615.446504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495535360:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573615.446519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.446522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573615.446524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.446528:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1831 00000100:00080000:0.0:1788573615.446531:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.446534:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573615.446538:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573615.446547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495535616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573615.446554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.446557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573615.446559:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573615.446560:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573615.446564:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1831 00000100:00080000:0.0:1788573615.446572:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.446575:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573615.446579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573615.446588:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495535744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573615.446603:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff93a2460d0a80 x1875453495535360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573615.446619:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495535360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573615.446636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573615.446639:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573615.446642:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495535360:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573615.446661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495535488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573615.446679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573615.446687:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff93a2460d0700 x1875453495535488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573615.446695:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495535488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573615.446700:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573615.446702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573615.446704:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495535488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573615.446708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495535616:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573615.446718:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573615.446726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff93a2460d3b80 x1875453495535616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573615.446738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495535616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573615.446746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573615.446750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573615.446753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495535616:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573615.446767:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495535744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573615.446791:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573615.446804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff93a2460d0000 x1875453495535744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573615.446813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495535744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573615.446819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573615.446821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573615.446846:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495535744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573616.406336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573616.406389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573617.430325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573617.430356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573618.454320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573618.454434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573619.478356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573619.478440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573620.502359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573620.502452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573620.566290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.566298:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573620.566302:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.566310:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1837 00000100:00080000:0.0:1788573620.566318:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.566325:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573620.566356:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495535872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573620.566381:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.566383:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573620.566385:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.566388:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1837 00000100:00080000:0.0:1788573620.566390:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.566392:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573620.566400:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573620.566407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495536000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573620.566414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.566416:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573620.566418:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.566421:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1837 00000100:00080000:0.0:1788573620.566422:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.566424:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573620.566427:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573620.566448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495536128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573620.566457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.566460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573620.566462:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.566465:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1837 00000100:00080000:0.0:1788573620.566467:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.566471:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573620.566475:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573620.566480:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495535872:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573620.566484:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495536256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573620.566573:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573620.566621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff93a2460d1500 x1875453495535872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573620.566639:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495535872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573620.566648:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573620.566652:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573620.566655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495535872:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573620.566673:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573620.566693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573620.566702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff93a2460d2680 x1875453495536000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573620.566710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495536000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573620.566715:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573620.566718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573620.566720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573620.566724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536128:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573620.566735:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573620.566741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff93a2460d0380 x1875453495536128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573620.566749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495536128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573620.566768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573620.566771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573620.566773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573620.566801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495536256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573620.566814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573620.566821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff93a2460d3480 x1875453495536256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573620.566835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495536256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573620.566850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573620.566852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573620.566854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495536256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573621.526331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573621.526386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573622.550340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573622.550382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573623.574339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573623.574380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573624.598328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573624.598392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573625.622323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573625.622414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573625.686349:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.686360:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573625.686364:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.686374:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1842 00000100:00080000:2.0:1788573625.686384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.686391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573625.686434:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495536384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.686460:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.686464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573625.686465:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.686469:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1842 00000100:00080000:2.0:1788573625.686472:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.686475:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573625.686497:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573625.686509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495536512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.686519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.686522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573625.686524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.686527:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1842 00000100:00080000:2.0:1788573625.686530:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.686536:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573625.686538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495536384:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573625.686540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573625.686552:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495536640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.686561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.686564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573625.686566:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.686568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1842 00000100:00080000:2.0:1788573625.686571:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.686573:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573625.686578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573625.686587:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495536768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573625.686611:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573625.686657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff93a24940b800 x1875453495536384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.686693:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495536384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.686704:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.686707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573625.686711:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495536384:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573625.686733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573625.686754:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573625.686765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff93a249408e00 x1875453495536512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.686783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495536512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.686791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.686793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573625.686796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573625.686800:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536640:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573625.686817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573625.686825:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff93a24940a680 x1875453495536640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.686834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495536640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.686840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.686842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573625.686844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495536640:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573625.686848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495536768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573625.686860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573625.686867:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff93a249409c00 x1875453495536768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.686875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495536768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.686891:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.686894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573625.686897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495536768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573626.646344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573626.646419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573627.670329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573627.670406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573628.694306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573628.694340:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573629.718351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573629.718417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573630.742303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573630.742361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573630.806356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.806364:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573630.806368:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.806376:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1847 00000100:00080000:0.0:1788573630.806385:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.806408:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573630.806444:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495536896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573630.806472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.806475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573630.806476:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.806480:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1847 00000100:00080000:0.0:1788573630.806483:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.806486:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573630.806495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573630.806506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495537024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573630.806514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.806518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573630.806519:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495536896:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573630.806520:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.806529:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1847 00000100:00080000:0.0:1788573630.806532:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.806534:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573630.806537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573630.806546:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495537152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573630.806553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.806556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573630.806563:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.806566:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1847 00000100:00080000:0.0:1788573630.806570:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.806573:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573630.806577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573630.806588:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495537280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573630.806588:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573630.806645:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff93a2460d2a00 x1875453495536896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.806660:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495536896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.806668:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.806671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573630.806674:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495536896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573630.806692:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573630.806709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573630.806718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff93a2460d1180 x1875453495537024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.806726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495537024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.806737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.806740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573630.806742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573630.806746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537152:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573630.806761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573630.806768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff93a2460d2300 x1875453495537152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.806775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495537152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.806780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.806782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573630.806784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537152:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573630.806787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495537280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573630.806797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573630.806803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff93a2460d1f80 x1875453495537280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.806810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495537280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.806814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.806816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573630.806825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495537280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573631.766299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573631.766365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573632.790318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573632.790392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573633.814329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573633.814413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573634.838334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573634.838392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573635.862358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573635.862436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573635.926311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573635.926322:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573635.926348:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573635.926360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1852 00000100:00080000:2.0:1788573635.926374:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573635.926391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573635.926437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495537408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573635.926463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573635.926467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573635.926469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573635.926473:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1852 00000100:00080000:2.0:1788573635.926476:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573635.926479:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573635.926489:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573635.926499:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495537536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573635.926507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573635.926510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573635.926512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573635.926517:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1852 00000100:00100000:3.0:1788573635.926519:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495537408:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573635.926520:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573635.926526:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573635.926545:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573635.926557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495537664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573635.926576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573635.926579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573635.926581:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573635.926584:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1852 00000100:00080000:2.0:1788573635.926588:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573635.926591:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573635.926595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573635.926606:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495537792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573635.926626:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573635.926682:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff93a24940ad80 x1875453495537408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.926708:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495537408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.926722:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.926726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573635.926730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495537408:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573635.926754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573635.926774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573635.926784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff93a249408000 x1875453495537536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.926792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495537536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.926798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.926801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573635.926810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573635.926815:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537664:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573635.926831:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573635.926842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff93a249409180 x1875453495537664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.926856:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495537664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.926865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.926868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573635.926872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495537664:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573635.926879:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495537792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573635.926899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573635.926908:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff93a249408a80 x1875453495537792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.926917:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495537792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.926922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.926924:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573635.926926:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495537792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573636.886368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573636.886432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573637.910357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573637.910432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573638.934353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573638.934396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573639.958363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573639.958433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573640.982355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573640.982409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573641.046359:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.046375:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573641.046381:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.046404:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1857 00000100:00080000:0.0:1788573641.046418:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.046428:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573641.046516:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495537920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573641.046557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.046584:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573641.046587:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.046594:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1857 00000100:00080000:0.0:1788573641.046600:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.046605:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573641.046612:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495537920:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573641.046621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573641.046642:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495538048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573641.046658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.046662:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573641.046664:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.046670:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1857 00000100:00080000:0.0:1788573641.046685:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573641.046690:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573641.046694:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573641.046702:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573641.046718:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495538176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573641.046730:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.046735:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573641.046737:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.046745:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1857 00000100:00100000:3.0:1788573641.046745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff93a2460d1f80 x1875453495537920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573641.046753:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.046761:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573641.046769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495537920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573641.046777:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788573641.046786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573641.046794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573641.046798:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495538304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573641.046803:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495537920:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573641.046842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573641.046865:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573641.046875:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff93a2460d2300 x1875453495538048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573641.046884:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495538048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573641.046890:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573641.046893:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573641.046895:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573641.046899:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538176:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573641.046924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573641.046931:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff93a2460d1180 x1875453495538176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573641.046939:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495538176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573641.046945:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573641.046947:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573641.046949:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538176:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573641.046952:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495538304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573641.046963:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573641.046969:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff93a2460d2a00 x1875453495538304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573641.046976:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495538304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573641.046982:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573641.046984:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573641.046987:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495538304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573642.006375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573642.006476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573643.030321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573643.030376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573644.054360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573644.054457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573645.078310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573645.078358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573646.102304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573646.102404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573646.166297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573646.166305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573646.166308:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573646.166317:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1862 00000100:00080000:0.0:1788573646.166326:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573646.166332:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573646.166363:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495538432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573646.166409:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573646.166413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573646.166415:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573646.166420:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1862 00000100:00080000:0.0:1788573646.166423:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573646.166426:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573646.166435:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573646.166437:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495538432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573646.166447:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495538560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573646.166474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573646.166476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573646.166478:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573646.166481:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1862 00000100:00080000:0.0:1788573646.166485:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573646.166487:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573646.166492:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573646.166501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495538688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573646.166508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573646.166513:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573646.166514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573646.166517:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573646.166521:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1862 00000100:00080000:0.0:1788573646.166523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573646.166526:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573646.166530:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573646.166538:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495538816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573646.166554:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff93a2460d3480 x1875453495538432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573646.166569:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495538432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573646.166590:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573646.166595:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573646.166600:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495538432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573646.166623:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573646.166649:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573646.166663:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff93a2460d0380 x1875453495538560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573646.166673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495538560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573646.166679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573646.166682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573646.166684:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573646.166688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538688:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573646.166703:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573646.166710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff93a2460d2680 x1875453495538688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573646.166729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495538688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573646.166735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573646.166737:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573646.166739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495538688:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573646.166743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495538816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573646.166756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573646.166763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff93a2460d1500 x1875453495538816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573646.166771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495538816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573646.166776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573646.166784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573646.166787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495538816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573647.126347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573647.126412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573648.150317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573648.150424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573649.174328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573649.174389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573650.198307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573650.198366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573651.222338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573651.222392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573651.286340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573651.286348:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573651.286352:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573651.286361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1867 00000100:00080000:2.0:1788573651.286369:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573651.286376:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573651.286410:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495538944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573651.286439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573651.286444:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573651.286447:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573651.286461:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1867 00000100:00080000:2.0:1788573651.286466:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573651.286471:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495538944:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573651.286475:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573651.286500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573651.286517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495539072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573651.286528:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573651.286542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573651.286551:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573651.286554:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573651.286560:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1867 00000100:00080000:2.0:1788573651.286566:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573651.286571:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573651.286577:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573651.286578:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff93a24940aa00 x1875453495538944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573651.286592:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495539200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573651.286605:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495538944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573651.286605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573651.286615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573651.286617:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573651.286627:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1867 00000100:00080000:0.0:1788573651.286632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573651.286633:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573651.286640:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:0.0:1788573651.286644:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000008:00100000:2.0:1788573651.286646:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573651.286654:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495538944:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573651.286658:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495539328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573651.286679:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573651.286700:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573651.286710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff93a249408700 x1875453495539072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573651.286719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495539072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573651.286725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573651.286727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573651.286729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573651.286733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539200:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573651.286744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573651.286750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff93a249409880 x1875453495539200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573651.286770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495539200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573651.286776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573651.286778:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573651.286780:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539200:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573651.286784:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495539328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573651.286796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573651.286803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff93a249409500 x1875453495539328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573651.286810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495539328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573651.286815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573651.286817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573651.286819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495539328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573652.246319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573652.246382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573653.270366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573653.270444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573654.294332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573654.294408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573655.318344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573655.318392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573656.342322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573656.342446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573656.406357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.406369:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573656.406373:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.406385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1872 00000100:00080000:3.0:1788573656.406398:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.406405:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573656.406449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495539456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573656.406478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.406481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573656.406482:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.406485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1872 00000100:00080000:3.0:1788573656.406488:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.406490:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573656.406501:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573656.406512:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495539584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573656.406519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.406522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573656.406523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.406527:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1872 00000100:00080000:3.0:1788573656.406529:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.406533:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573656.406540:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495539456:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788573656.406557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573656.406573:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495539712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573656.406582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.406584:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573656.406586:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.406589:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1872 00000100:00080000:3.0:1788573656.406593:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.406596:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573656.406600:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573656.406607:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495539840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573656.406617:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573656.406657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff93a25064d180 x1875453495539456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573656.406690:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495539456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573656.406703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573656.406708:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573656.406722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495539456:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573656.406733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573656.406757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573656.406768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff93a25064c000 x1875453495539584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573656.406777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495539584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573656.406783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573656.406786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573656.406788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573656.406817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539712:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573656.406842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573656.406855:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff93a25064f480 x1875453495539712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573656.406866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495539712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573656.406882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573656.406884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573656.406887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495539712:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573656.406891:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495539840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573656.406908:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573656.406916:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff93a25064fb80 x1875453495539840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573656.406923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495539840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573656.406928:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573656.406930:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573656.406932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495539840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573657.366348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573657.366407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573658.390327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573658.390364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573659.414323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573659.414380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573660.438379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573660.438480:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573661.462377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573661.463144:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573661.526334:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.526346:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573661.526352:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.526362:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1877 00000100:00080000:3.0:1788573661.526373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.526381:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573661.526419:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495539968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.526463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.526466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573661.526467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.526471:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1877 00000100:00080000:3.0:1788573661.526474:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.526477:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573661.526485:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573661.526496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495540096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.526505:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573661.526507:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495539968:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573661.526509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573661.526512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.526520:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1877 00000100:00080000:3.0:1788573661.526523:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.526527:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573661.526533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573661.526547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495540224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.526559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.526563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573661.526565:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.526569:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1877 00000100:00080000:3.0:1788573661.526580:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.526584:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573661.526589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573661.526600:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495540352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573661.526617:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573661.526664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540224:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573661.526696:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495540352:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573662.486695:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573662.486929:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573663.510346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573663.510959:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573664.534508:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573664.534819:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573664.598483:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788573664.598528:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573664.598547:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573664.598589:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573664.598601:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573664.598610:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788573664.598712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff93a25064e680 x1875453495540096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.598739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495540096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.598753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.598759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573664.598763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573664.598772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff93a25064ed80 x1875453495540224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.598784:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495540224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.598792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.598796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573664.598800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540224:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573664.598808:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff93a25064dc00 x1875453495540352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.598820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495540352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.598829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.598840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573664.598844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495540352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573664.598852:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff93a25064d880 x1875453495539968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.598874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495539968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.598882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.598885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573664.598889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495539968:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573665.558564:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573665.559170:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573666.582347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573666.582441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573666.646408:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573666.646423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573666.646429:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573666.646443:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1883 00000100:00080000:0.0:1788573666.646458:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573666.646468:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573666.646514:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495540480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573666.646573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573666.646577:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573666.646580:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573666.646585:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1883 00000100:00080000:0.0:1788573666.646589:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573666.646595:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573666.646630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573666.646659:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495540608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573666.646671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573666.646675:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573666.646677:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573666.646682:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1883 00000100:00080000:0.0:1788573666.646686:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573666.646691:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573666.646697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573666.646709:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495540736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573666.646720:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573666.646724:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573666.646726:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573666.646731:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1883 00000100:00080000:0.0:1788573666.646735:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573666.646739:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573666.646744:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573666.646756:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495540864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573666.646911:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495540480:192.168.202.102@tcp:250:kworker.0 00000800:00000100:3.0:1788573666.647027:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573666.647088:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff93a2460d0000 x1875453495540480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573666.647113:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495540480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573666.647126:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573666.647132:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573666.647137:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495540480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573666.647146:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573666.647183:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573666.647323:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff93a2460d3b80 x1875453495540608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573666.647338:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495540608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573666.647364:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573666.647368:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573666.647372:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573666.647379:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540736:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573666.647406:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573666.647419:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff93a2460d0700 x1875453495540736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573666.647433:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495540736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573666.647443:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573666.647446:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573666.647450:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495540736:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573666.647456:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495540864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573666.647480:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573666.647492:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff93a2460d0a80 x1875453495540864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573666.647503:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495540864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573666.647512:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573666.647514:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573666.647534:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495540864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573667.606342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573667.606397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573668.630374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573668.630470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573669.654314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573669.654401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573670.678384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573670.678447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573671.702289:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573671.702407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573671.766330:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573671.766343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573671.766348:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573671.766366:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1888 00000100:00080000:3.0:1788573671.766383:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573671.766393:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573671.766455:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495540992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573671.766484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573671.766487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573671.766489:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573671.766492:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1888 00000100:00080000:3.0:1788573671.766496:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573671.766499:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573671.766509:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573671.766526:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495541120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573671.766533:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495540992:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573671.766545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573671.766550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573671.766552:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573671.766558:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1888 00000100:00080000:3.0:1788573671.766563:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573671.766592:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573671.766597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573671.766608:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495541248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573671.766617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573671.766622:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573671.766623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573671.766627:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573671.766637:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1888 00000100:00080000:3.0:1788573671.766642:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573671.766646:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573671.766650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573671.766659:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495541376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573671.766686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff93a25064ea00 x1875453495540992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573671.766706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495540992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573671.766716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573671.766719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573671.766722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495540992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573671.766747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573671.766779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573671.766803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff93a25064d880 x1875453495541120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573671.766815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495541120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573671.766821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573671.766824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573671.766836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573671.766847:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541248:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788573671.766884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573671.766898:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff93a25064dc00 x1875453495541248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573671.766910:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495541248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573671.766918:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573671.766922:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573671.766925:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541248:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788573671.766931:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495541376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788573671.766964:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573671.766977:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff93a25064ed80 x1875453495541376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573671.766987:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495541376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573671.766993:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573671.766995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573671.766998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495541376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573672.726348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573672.726422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573673.750316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573673.750350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573674.774341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573674.774402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573675.798330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573675.798417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573676.822309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573676.822362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573676.886329:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573676.886338:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573676.886341:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573676.886350:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1893 00000100:00080000:3.0:1788573676.886358:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573676.886364:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573676.886400:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495541504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573676.886435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573676.886439:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573676.886457:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573676.886464:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1893 00000100:00080000:3.0:1788573676.886469:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573676.886475:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573676.886482:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495541504:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788573676.886486:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573676.886501:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495541632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573676.886510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573676.886513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573676.886515:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573676.886518:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1893 00000100:00080000:3.0:1788573676.886523:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573676.886525:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573676.886529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573676.886538:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495541760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573676.886544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573676.886552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573676.886555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573676.886557:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573676.886561:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1893 00000100:00080000:3.0:1788573676.886563:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573676.886566:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573676.886569:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573676.886585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495541888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573676.886593:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573676] req@ffff93a25064e680 x1875453495541504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573676.886617:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495541504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573676.886629:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573676.886634:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573676.886648:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495541504:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573676.886688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573676.886708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573676.886717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573676] req@ffff93a25064fb80 x1875453495541632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573676.886738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495541632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573676.886744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573676.886746:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573676.886749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573676.886755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541760:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573676.886776:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573676.886788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573676] req@ffff93a25064f480 x1875453495541760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573676.886802:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495541760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573676.886811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573676.886814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573676.886817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495541760:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573676.886821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495541888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573676.886837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573676.886844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573676] req@ffff93a25064c000 x1875453495541888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573676.886851:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495541888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573676.886866:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573676.886868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573676.886870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495541888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573677.846313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573677.846361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573678.870326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573678.870457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573679.894315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573679.894367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573680.918336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573680.918400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573681.942328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573681.942417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573682.006345:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573682.006356:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573682.006361:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573682.006378:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1898 00000100:00080000:0.0:1788573682.006390:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573682.006397:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573682.006442:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495542016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573682.006471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573682.006474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573682.006476:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573682.006479:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1898 00000100:00080000:0.0:1788573682.006482:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573682.006485:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573682.006495:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573682.006505:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495542144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573682.006513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573682.006515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573682.006517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573682.006522:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1898 00000100:00100000:1.0:1788573682.006524:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495542016:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573682.006566:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573682.006571:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573682.006576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573682.006587:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495542272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573682.006596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573682.006599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573682.006600:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573682.006604:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1898 00000100:00080000:0.0:1788573682.006606:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573682.006609:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573682.006611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573682.006618:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573682.006622:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495542400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573682.006667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff93a2460d0a80 x1875453495542016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.006684:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495542016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.006693:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.006696:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573682.006699:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495542016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573682.006723:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573682.006739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573682.006747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff93a2460d0700 x1875453495542144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.006755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495542144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.006761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.006764:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573682.006766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573682.006778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542272:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573682.006794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573682.006801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff93a2460d3b80 x1875453495542272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.006808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495542272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.006814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.006816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573682.006819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542272:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573682.006822:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495542400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573682.006832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573682.006838:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff93a2460d0000 x1875453495542400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.006845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495542400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.006850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.006852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573682.006854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495542400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573682.966336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573682.966406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573683.990320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573683.990383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573685.014315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573685.014381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573686.038352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573686.038409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573687.062319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573687.062448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573687.126404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573687.126445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573687.126450:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573687.126476:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1903 00000100:00080000:2.0:1788573687.126486:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573687.126493:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573687.126535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495542528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573687.126573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573687.126576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573687.126578:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573687.126581:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1903 00000100:00080000:2.0:1788573687.126584:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573687.126587:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573687.126595:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573687.126608:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495542656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573687.126616:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573687.126619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573687.126620:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573687.126623:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1903 00000100:00080000:2.0:1788573687.126626:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573687.126629:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573687.126632:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573687.126646:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495542784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573687.126650:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495542528:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573687.126671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573687.126675:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573687.126676:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573687.126680:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1903 00000100:00080000:2.0:1788573687.126684:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573687.126687:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573687.126691:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573687.126700:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495542912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573687.126737:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573687.126799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff93a24940b100 x1875453495542528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.126834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495542528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.126844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.126848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573687.126851:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495542528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573687.126860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573687.126887:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573687.126902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff93a24940bb80 x1875453495542656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.126915:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495542656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.126921:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.126924:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573687.126926:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573687.126958:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542784:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573687.126983:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573687.126994:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff93a24940b480 x1875453495542784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.127003:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495542784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.127018:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.127021:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573687.127023:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495542784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573687.127028:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495542912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573687.127043:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573687.127052:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff93a249409500 x1875453495542912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.127064:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495542912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.127073:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.127077:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573687.127080:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495542912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573688.086341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573688.086444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573689.110302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573689.110377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573690.134351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573690.134470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573691.158328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573691.158465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573692.182326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573692.182407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573692.246333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573692.246345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573692.246364:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573692.246371:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1908 00000100:00080000:0.0:1788573692.246381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573692.246386:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573692.246413:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495543040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573692.246434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573692.246437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573692.246439:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573692.246442:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1908 00000100:00080000:0.0:1788573692.246444:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573692.246446:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573692.246453:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573692.246462:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495543168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573692.246470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573692.246472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573692.246473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573692.246479:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1908 00000100:00100000:1.0:1788573692.246480:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495543040:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573692.246481:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573692.246486:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573692.246494:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573692.246500:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495543296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573692.246517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573692.246520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573692.246521:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573692.246524:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1908 00000100:00080000:0.0:1788573692.246543:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573692.246546:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573692.246550:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573692.246557:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573692.246560:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495543424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573692.246607:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff93a2460d1500 x1875453495543040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.246624:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495543040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.246632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.246635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573692.246639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495543040:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573692.246664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573692.246687:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573692.246700:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff93a2460d2680 x1875453495543168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.246714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495543168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.246723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.246727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573692.246730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573692.246735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543296:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573692.246751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573692.246759:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff93a2460d0380 x1875453495543296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.246767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495543296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.246772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.246774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573692.246776:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543296:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573692.246781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495543424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573692.246799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573692.246810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff93a2460d3480 x1875453495543424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.246830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495543424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.246836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.246838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573692.246840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495543424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573693.206306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573693.206388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573694.230353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573694.230418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573695.254394:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573695.254462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573696.278347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573696.278407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573697.302308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573697.302424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573697.366327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573697.366341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573697.366345:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573697.366353:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1913 00000100:00080000:0.0:1788573697.366363:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573697.366369:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573697.366404:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495543552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573697.366429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573697.366432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573697.366434:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573697.366438:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1913 00000100:00080000:0.0:1788573697.366454:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573697.366457:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573697.366467:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573697.366479:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495543680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573697.366485:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495543552:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573697.366490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573697.366494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573697.366495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573697.366499:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1913 00000100:00080000:0.0:1788573697.366501:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573697.366504:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573697.366508:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573697.366515:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495543808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573697.366523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573697.366526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573697.366527:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573697.366529:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1913 00000100:00080000:0.0:1788573697.366531:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573697.366534:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573697.366537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573697.366543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495543936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573697.366561:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573697.366604:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff93a2460d2a00 x1875453495543552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573697.366618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495543552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573697.366637:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573697.366640:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573697.366643:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495543552:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573697.366662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573697.366680:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573697.366689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff93a2460d1180 x1875453495543680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573697.366697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495543680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573697.366702:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573697.366705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573697.366707:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573697.366710:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543808:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573697.366721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573697.366726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff93a2460d2300 x1875453495543808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573697.366733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495543808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573697.366739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573697.366740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573697.366742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495543808:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573697.366751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495543936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573697.366764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573697.366771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff93a2460d1f80 x1875453495543936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573697.366778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495543936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573697.366783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573697.366784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573697.366786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495543936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573698.326365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573698.326436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573699.350306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573699.350366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573700.374316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573700.374369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573701.398311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573701.398356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573702.422320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573702.422381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573702.486313:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573702.486321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573702.486325:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573702.486333:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1918 00000100:00080000:1.0:1788573702.486342:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573702.486349:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573702.486379:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495544064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573702.486401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573702.486404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573702.486409:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573702.486413:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1918 00000100:00080000:1.0:1788573702.486416:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573702.486430:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573702.486439:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573702.486443:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495544064:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573702.486451:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495544192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573702.486479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573702.486482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573702.486483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573702.486497:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1918 00000100:00080000:1.0:1788573702.486503:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573702.486505:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573702.486509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573702.486512:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573702.486525:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495544320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573702.486537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573702.486541:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573702.486543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573702.486549:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff93a258da0a80 x1875453495544064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573702.486550:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1918 00000100:00080000:1.0:1788573702.486565:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573702.486573:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573702.486579:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573702.486580:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495544064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573702.486591:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788573702.486594:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495544448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573702.486599:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573702.486607:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495544064:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573702.486635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573702.486654:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573702.486672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff93a258da0700 x1875453495544192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573702.486681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495544192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573702.486686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573702.486689:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573702.486691:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573702.486695:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544320:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573702.486706:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573702.486712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff93a258da1c00 x1875453495544320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573702.486719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495544320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573702.486724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573702.486726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573702.486728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544320:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573702.486731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495544448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573702.486741:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573702.486747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff93a258da1880 x1875453495544448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573702.486761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495544448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573702.486767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573702.486769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573702.486771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495544448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573703.446370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573703.446375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573704.470327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573704.470383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573705.494369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573705.494436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573706.518343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573706.518387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573707.542368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573707.542437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573707.606337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.606346:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573707.606350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.606360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1924 00000100:00080000:3.0:1788573707.606372:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.606379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573707.606415:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495544576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573707.606437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.606440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573707.606441:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.606445:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1924 00000100:00080000:3.0:1788573707.606461:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.606465:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573707.606473:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573707.606483:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495544704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573707.606490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.606492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573707.606494:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.606499:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1924 00000100:00080000:3.0:1788573707.606502:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.606506:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573707.606511:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573707.606523:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495544832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573707.606535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.606539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573707.606542:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.606547:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1924 00000100:00080000:3.0:1788573707.606550:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.606553:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573707.606557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573707.606564:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495544576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573707.606586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495544960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573707.606651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573707.606711:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff93a25064d180 x1875453495544576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573707.606733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495544576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573707.606744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573707.606748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573707.606752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495544576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573707.606759:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573707.606777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573707.606785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff93a25064f100 x1875453495544704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573707.606812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495544704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573707.606822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573707.606826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573707.606830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573707.606837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544832:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573707.606861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573707.606874:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff93a25064ce00 x1875453495544832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573707.606883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495544832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573707.606889:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573707.606908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573707.606911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495544832:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573707.606940:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495544960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573707.606957:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573707.606965:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff93a25064e300 x1875453495544960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573707.606972:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495544960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573707.606999:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573707.607001:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573707.607004:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495544960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573708.566350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573708.566426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573709.590310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573709.590394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573710.614355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573710.614434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573711.638388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573711.638458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573712.662344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573712.662375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573712.726338:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573712.726347:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573712.726350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573712.726359:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1929 00000100:00080000:1.0:1788573712.726383:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573712.726389:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573712.726420:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495545088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573712.726442:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573712.726445:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573712.726446:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573712.726449:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1929 00000100:00080000:1.0:1788573712.726451:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573712.726454:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573712.726461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573712.726469:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495545216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573712.726476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573712.726479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573712.726480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573712.726483:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1929 00000100:00080000:1.0:1788573712.726486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573712.726489:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573712.726492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573712.726498:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495545344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573712.726505:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573712.726512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573712.726514:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573712.726517:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1929 00000100:00080000:1.0:1788573712.726520:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573712.726528:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573712.726532:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495545088:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788573712.726533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573712.726545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495545472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573712.726636:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573712.726710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff93a258da3100 x1875453495545088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573712.726737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495545088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573712.726752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573712.726758:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573712.726796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495545088:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573712.726809:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573712.726848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573712.726865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff93a258da2a00 x1875453495545216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573712.726879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495545216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573712.726904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573712.726909:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573712.726913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573712.726921:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545344:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573712.726950:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573712.726964:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff93a258da3480 x1875453495545344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573712.726978:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495545344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573712.726988:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573712.726992:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573712.726996:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545344:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573712.727034:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495545472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573712.727059:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573712.727073:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff93a258da0380 x1875453495545472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573712.727087:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495545472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573712.727097:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573712.727102:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573712.727114:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495545472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573713.686411:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573713.686591:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573714.710344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573714.710415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573715.734335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573715.734409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573716.758348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573716.758415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573717.782373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573717.782452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573717.846397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573717.846410:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573717.846414:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573717.846426:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1934 00000100:00080000:1.0:1788573717.846439:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573717.846446:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573717.846494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495545600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573717.846525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573717.846529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573717.846530:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573717.846534:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1934 00000100:00080000:1.0:1788573717.846536:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573717.846539:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573717.846548:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573717.846558:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495545728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573717.846584:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495545600:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573717.846597:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573717.846600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573717.846601:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573717.846606:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1934 00000100:00080000:1.0:1788573717.846609:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573717.846613:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573717.846618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573717.846629:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495545856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573717.846637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573717.846645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573717.846659:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573717.846663:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1934 00000100:00080000:1.0:1788573717.846666:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573717.846669:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573717.846674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573717.846678:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573717.846689:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495545984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573717.846737:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff93a258da2300 x1875453495545600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573717.846755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495545600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573717.846765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573717.846768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573717.846772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495545600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573717.846795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573717.846813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573717.846821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff93a258da0000 x1875453495545728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573717.846829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495545728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573717.846836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573717.846838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573717.846847:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573717.846851:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545856:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573717.846871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573717.846882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff93a258da1180 x1875453495545856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573717.846895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495545856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573717.846904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573717.846907:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573717.846912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495545856:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573717.846919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495545984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573717.846937:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573717.846945:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff93a258da0380 x1875453495545984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573717.846953:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495545984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573717.846959:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573717.846961:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573717.846964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495545984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573718.806364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573718.806431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573719.830304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573719.830405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573720.854345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573720.854409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573721.878345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573721.878411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573722.902355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573722.902450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573722.966421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.966433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573722.966438:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.966449:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1939 00000100:00080000:2.0:1788573722.966478:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.966486:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573722.966535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495546112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573722.966608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.966611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573722.966613:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.966616:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1939 00000100:00080000:2.0:1788573722.966620:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.966624:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573722.966639:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573722.966652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495546112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573722.966653:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495546240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573722.966667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.966670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573722.966671:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.966673:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1939 00000100:00080000:2.0:1788573722.966676:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.966698:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573722.966703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573722.966711:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495546368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573722.966719:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.966722:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573722.966723:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.966725:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1939 00000100:00080000:2.0:1788573722.966729:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573722.966732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573722.966736:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573722.966740:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573722.966748:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495546496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573722.966807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff93a249409880 x1875453495546112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573722.966839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495546112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573722.966849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573722.966853:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573722.966857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495546112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573722.966885:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573722.966905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573722.966915:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff93a249408700 x1875453495546240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573722.966924:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495546240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573722.966930:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573722.966933:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573722.966935:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573722.966939:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546368:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573722.966952:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573722.966958:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff93a24940aa00 x1875453495546368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573722.966966:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495546368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573722.966983:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573722.966985:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573722.966988:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546368:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573722.966992:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495546496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573722.967008:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573722.967015:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff93a249408a80 x1875453495546496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573722.967023:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495546496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573722.967029:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573722.967031:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573722.967033:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495546496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573723.926362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573723.926478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573724.950362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573724.950483:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573725.974301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573725.974419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573726.998365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573726.998432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573728.022439:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573728.022692:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573728.086411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.086420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573728.086424:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.086433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1944 00000100:00080000:3.0:1788573728.086441:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.086462:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573728.086495:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495546624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573728.086522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.086526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573728.086543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.086548:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1944 00000100:00080000:3.0:1788573728.086553:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.086558:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573728.086569:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573728.086583:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495546624:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573728.086586:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495546752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573728.086598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.086600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573728.086602:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.086605:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1944 00000100:00080000:3.0:1788573728.086608:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.086611:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573728.086615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573728.086624:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495546880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573728.086631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.086634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573728.086635:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.086638:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1944 00000100:00080000:3.0:1788573728.086640:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.086642:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573728.086645:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573728.086658:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495547008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573728.086691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573728.086717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546880:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573728.086732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495547008:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573729.046581:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573729.047264:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573730.070520:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573730.070595:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573731.094737:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573731.095907:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:3.0:1788573731.158624:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788573731.158692:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573731.158717:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788573731.158774:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788573731.158786:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788573731.158798:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788573731.158853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff93a25064d500 x1875453495546752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.158929:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495546752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.158946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.158952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573731.158956:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573731.158967:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff93a25064c700 x1875453495546880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.158983:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495546880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.158991:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.158995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573731.158998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495546880:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573731.159007:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff93a25064ca80 x1875453495547008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.159020:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495547008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.159030:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.159034:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573731.159038:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495547008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573731.159046:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff93a25064c380 x1875453495546624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.159070:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495546624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.159079:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.159084:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573731.159088:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495546624:192.168.202.102@tcp:250:kworker.0 00010000:00010000:0.0:1788573732.118716:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573732.119015:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573733.142337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573733.142437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573733.206319:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573733.206329:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573733.206334:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573733.206358:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1949 00000100:00080000:3.0:1788573733.206369:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573733.206377:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573733.206410:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495547136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573733.206441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573733.206445:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573733.206447:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573733.206450:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1949 00000100:00080000:3.0:1788573733.206454:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573733.206458:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573733.206468:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573733.206484:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495547264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573733.206504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495547136:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573733.206516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573733.206521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573733.206524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573733.206530:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1949 00000100:00080000:3.0:1788573733.206535:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573733.206539:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573733.206545:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573733.206559:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495547392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573733.206571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573733.206576:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573733.206578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573733.206582:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573733.206587:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1949 00000100:00080000:3.0:1788573733.206592:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573733.206596:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573733.206602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573733.206615:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495547520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573733.206653:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff93a25064df80 x1875453495547136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.206678:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495547136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.206691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.206696:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573733.206700:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495547136:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573733.206732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573733.206758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573733.206773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff93a25064c380 x1875453495547264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.206787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495547264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.206796:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.206808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573733.206812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573733.206820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547392:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573733.206843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573733.206854:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff93a25064ca80 x1875453495547392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.206867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495547392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.206876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.206880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573733.206883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547392:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573733.206889:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495547520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573733.206905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573733.206914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff93a25064c700 x1875453495547520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.206925:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495547520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.206934:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.206937:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573733.206940:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495547520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573734.166356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573734.166427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573735.190298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573735.190368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573736.214320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573736.214384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573737.238316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573737.238384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573738.262329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573738.262422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573738.326293:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.326302:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573738.326306:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.326314:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1954 00000100:00080000:3.0:1788573738.326345:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.326358:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573738.326389:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495547648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.326411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.326414:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573738.326415:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.326418:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1954 00000100:00080000:3.0:1788573738.326420:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.326423:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573738.326431:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573738.326446:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495547776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.326457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.326461:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573738.326463:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.326468:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1954 00000100:00080000:3.0:1788573738.326473:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573738.326483:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495547648:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573738.326486:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573738.326493:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573738.326504:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495547904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.326516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.326524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573738.326526:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.326530:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1954 00000100:00080000:3.0:1788573738.326535:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.326538:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573738.326543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573738.326552:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495548032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573738.326581:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573738.326633:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff93a25064d500 x1875453495547648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.326653:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495547648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.326664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.326668:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573738.326671:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495547648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573738.326678:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573738.326695:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573738.326704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff93a25064e300 x1875453495547776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.326712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495547776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.326718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.326720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573738.326722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573738.326750:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547904:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573738.326763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573738.326770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff93a25064ce00 x1875453495547904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.326791:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495547904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.326796:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.326799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573738.326808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495547904:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573738.326812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495548032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573738.326828:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573738.326836:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff93a25064f100 x1875453495548032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.326844:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495548032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.326849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.326852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573738.326854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495548032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573739.286333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573739.286397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573740.310301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573740.310380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573741.334349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573741.334444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573742.358336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573742.358399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573743.382326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573743.382376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573743.446320:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.446330:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573743.446334:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.446343:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1959 00000100:00080000:3.0:1788573743.446351:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.446357:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573743.446391:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495548160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.446413:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.446416:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.446417:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.446420:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1959 00000100:00080000:3.0:1788573743.446422:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.446425:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573743.446433:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573743.446441:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495548288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573743.446452:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495548160:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573743.446467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.446469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.446471:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.446475:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1959 00000100:00080000:3.0:1788573743.446478:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.446481:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573743.446485:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573743.446494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495548416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.446501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.446504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.446505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.446507:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1959 00000100:00080000:3.0:1788573743.446510:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573743.446514:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573743.446515:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573743.446520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573743.446527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495548544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573743.446561:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff93a25064d180 x1875453495548160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573743.446576:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495548160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573743.446595:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573743.446598:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573743.446603:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495548160:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573743.446623:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573743.446642:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573743.446650:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff93a25064c000 x1875453495548288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573743.446658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495548288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573743.446664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573743.446666:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573743.446668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573743.446671:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548416:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573743.446682:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573743.446688:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff93a25064f480 x1875453495548416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573743.446695:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495548416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573743.446700:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573743.446702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573743.446703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548416:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573743.446713:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495548544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573743.446726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573743.446733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff93a25064fb80 x1875453495548544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573743.446740:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495548544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573743.446745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573743.446747:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573743.446748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495548544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573744.406337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573744.406416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573745.430303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573745.430335:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573746.454343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573746.454436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573747.478300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573747.478378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573748.502339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573748.502398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573748.566352:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.566368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573748.566374:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.566388:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1965 00000100:00080000:3.0:1788573748.566428:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.566438:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573748.566482:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495548672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573748.566511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.566515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573748.566516:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.566520:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1965 00000100:00080000:3.0:1788573748.566523:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.566527:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573748.566537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573748.566551:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495548800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573748.566563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.566570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788573748.566574:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495548672:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573748.566574:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.566594:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1965 00000100:00080000:3.0:1788573748.566598:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.566602:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573748.566609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573748.566627:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495548928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573748.566640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.566654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573748.566657:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573748.566664:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573748.566671:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1965 00000100:00080000:3.0:1788573748.566678:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.566683:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573748.566690:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573748.566706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff93a25064e680 x1875453495548672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573748.566711:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495549056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573748.566739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495548672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573748.566752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573748.566762:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573748.566766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495548672:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573748.566791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573748.566809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573748.566824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff93a25064ed80 x1875453495548800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573748.566846:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495548800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573748.566852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573748.566855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573748.566857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573748.566862:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548928:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573748.566878:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573748.566885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff93a25064dc00 x1875453495548928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573748.566894:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495548928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573748.566899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573748.566901:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573748.566903:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495548928:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573748.566907:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495549056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573748.566918:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573748.566924:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff93a25064d880 x1875453495549056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573748.566932:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495549056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573748.566946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573748.566948:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573748.566951:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495549056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573749.526316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573749.526386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573750.550345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573750.550452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573751.574303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573751.574398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573752.598279:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573752.598351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573753.622356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573753.622417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573753.686375:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573753.686391:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573753.686397:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573753.686423:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1970 00000100:00080000:3.0:1788573753.686437:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573753.686447:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573753.686509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495549184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573753.686549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573753.686553:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573753.686555:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573753.686560:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1970 00000100:00080000:3.0:1788573753.686565:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573753.686570:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573753.686582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573753.686599:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495549312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573753.686641:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495549184:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573753.686643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573753.686651:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573753.686654:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573753.686661:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1970 00000100:00080000:3.0:1788573753.686667:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573753.686672:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573753.686680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573753.686696:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495549440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573753.686708:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573753.686712:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573753.686716:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573753.686717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573753.686726:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1970 00000100:00080000:3.0:1788573753.686733:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573753.686738:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573753.686745:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573753.686759:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495549568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573753.686777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff93a25064ea00 x1875453495549184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.686806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495549184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.686818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.686821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573753.686824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495549184:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573753.686848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573753.686871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573753.686880:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff93a25064d880 x1875453495549312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.686887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495549312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.686893:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.686896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573753.686898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573753.686910:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549440:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573753.686926:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573753.686933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff93a25064dc00 x1875453495549440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.686969:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495549440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.686975:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.686978:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573753.686980:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573753.686985:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495549568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573753.687000:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573753.687007:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff93a25064ed80 x1875453495549568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.687014:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495549568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.687019:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.687021:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573753.687023:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495549568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573754.646345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573754.646433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573755.670329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573755.670419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573756.694354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573756.694402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573757.718325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573757.718389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573758.742340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573758.742404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573758.806342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.806351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573758.806355:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.806370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1975 00000100:00080000:2.0:1788573758.806379:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.806385:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573758.806417:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495549696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573758.806437:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.806440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573758.806455:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.806458:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1975 00000100:00080000:2.0:1788573758.806461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.806464:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573758.806473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573758.806488:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495549824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573758.806497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495549696:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573758.806499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.806506:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573758.806508:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.806511:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1975 00000100:00080000:2.0:1788573758.806514:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.806516:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573758.806519:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573758.806526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495549952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573758.806533:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.806535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573758.806536:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.806538:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1975 00000100:00080000:2.0:1788573758.806540:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.806542:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573758.806545:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573758.806557:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495550080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573758.806579:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573758.806621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff93a249409180 x1875453495549696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.806642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495549696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.806651:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.806654:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573758.806657:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495549696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573758.806677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573758.806700:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573758.806714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff93a249408000 x1875453495549824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.806728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495549824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.806736:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.806739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573758.806741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573758.806745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549952:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573758.806762:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573758.806769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff93a24940ad80 x1875453495549952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.806777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495549952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.806803:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.806807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573758.806811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495549952:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573758.806818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495550080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573758.806854:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573758.806864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff93a249409c00 x1875453495550080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.806872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495550080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.806883:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.806885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573758.806888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495550080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573759.766305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573759.766399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573760.790320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573760.790391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573761.814316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573761.814441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573762.838345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573762.838415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573763.862350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573763.862410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573763.926411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573763.926420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573763.926423:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573763.926433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1980 00000100:00080000:1.0:1788573763.926443:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573763.926450:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573763.926488:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495550208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573763.926521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573763.926536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573763.926537:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573763.926541:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1980 00000100:00080000:1.0:1788573763.926545:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573763.926549:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573763.926558:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573763.926569:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495550208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573763.926572:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495550336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573763.926583:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573763.926585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573763.926587:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573763.926590:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1980 00000100:00080000:1.0:1788573763.926593:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573763.926595:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573763.926599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573763.926607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495550464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573763.926614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573763.926617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573763.926618:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573763.926620:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1980 00000100:00080000:1.0:1788573763.926623:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573763.926627:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788573763.926630:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573763.926641:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573763.926651:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495550592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573763.926682:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff93a258da3480 x1875453495550208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573763.926705:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495550208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573763.926717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573763.926720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573763.926723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495550208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573763.926764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573763.926785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573763.926799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff93a258da2a00 x1875453495550336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573763.926812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495550336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573763.926822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573763.926832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573763.926835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573763.926839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550464:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573763.926857:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573763.926865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff93a258da3100 x1875453495550464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573763.926873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495550464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573763.926879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573763.926881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573763.926883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550464:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573763.926886:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495550592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573763.926896:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573763.926902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff93a258da1880 x1875453495550592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573763.926909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495550592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573763.926914:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573763.926916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573763.926918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495550592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573764.886296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573764.886406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573765.910347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573765.910489:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573766.934296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573766.934411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573767.958319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573767.958462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573768.982285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573768.982363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573769.046355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573769.046366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573769.046371:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573769.046382:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1985 00000100:00080000:3.0:1788573769.046411:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573769.046419:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573769.046455:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495550720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573769.046483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573769.046488:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573769.046490:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573769.046495:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1985 00000100:00080000:3.0:1788573769.046499:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573769.046505:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573769.046516:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573769.046535:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495550720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573769.046535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495550848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573769.046554:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573769.046558:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573769.046560:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573769.046566:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1985 00000100:00080000:3.0:1788573769.046570:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573769.046576:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573769.046582:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573769.046594:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495550976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573769.046607:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573769.046608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573769.046620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573769.046623:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573769.046629:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1985 00000100:00080000:3.0:1788573769.046668:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573769.046671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff93a25064e680 x1875453495550720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573769.046678:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573769.046692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495550720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573769.046694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788573769.046715:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573769.046717:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495551104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573769.046724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573769.046732:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495550720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573769.046760:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573769.046783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573769.046793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff93a25064fb80 x1875453495550848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573769.046802:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495550848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573769.046808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573769.046811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573769.046813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573769.046817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550976:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573769.046850:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573769.046864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff93a25064f480 x1875453495550976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573769.046879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495550976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573769.046889:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573769.046893:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573769.046896:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495550976:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573769.046901:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495551104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573769.046924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573769.046937:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff93a25064c000 x1875453495551104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573769.046952:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495551104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573769.046962:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573769.046966:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573769.046970:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495551104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573770.006400:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573770.006505:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573771.030361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573771.030423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573772.054365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573772.054388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573773.078352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573773.078411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573774.102359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573774.102426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573774.166368:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573774.166380:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573774.166384:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573774.166401:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1990 00000100:00080000:0.0:1788573774.166413:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573774.166421:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573774.166467:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495551232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573774.166503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573774.166508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573774.166510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573774.166554:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1990 00000100:00080000:0.0:1788573774.166559:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573774.166563:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573774.166569:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495551232:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573774.166576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573774.166593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495551360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573774.166603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573774.166606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573774.166608:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573774.166611:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1990 00000100:00080000:0.0:1788573774.166614:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573774.166617:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573774.166621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573774.166630:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495551488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573774.166638:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573774.166641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573774.166642:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573774.166645:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1990 00000100:00080000:0.0:1788573774.166649:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573774.166651:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573774.166654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573774.166661:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495551616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573774.166677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573774.166726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff93a2460d1f80 x1875453495551232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573774.166744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495551232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573774.166753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573774.166756:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573774.166760:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495551232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573774.166784:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495551360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573774.166802:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573774.166810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff93a2460d2300 x1875453495551360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573774.166819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495551360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573774.166824:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573774.166827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573774.166829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495551360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573774.166832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495551488:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573774.166845:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573774.166851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff93a2460d1180 x1875453495551488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573774.166870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495551488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573774.166876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573774.166878:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573774.166880:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495551488:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573774.166884:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495551616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573774.166897:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573774.166903:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff93a2460d2a00 x1875453495551616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573774.166910:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495551616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573774.166915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573774.166917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573774.166919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495551616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573775.126364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573775.126425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573776.150379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573776.150437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573777.174299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573777.174407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573778.198331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573778.198420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573779.222305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573779.222368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573779.286322:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.286331:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573779.286334:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.286343:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 1995 00000100:00080000:3.0:1788573779.286351:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.286357:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573779.286388:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495551744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573779.286410:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.286413:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573779.286414:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.286417:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1995 00000100:00080000:3.0:1788573779.286419:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.286422:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573779.286430:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573779.286454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495551872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573779.286457:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495551744:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573779.286476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.286479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573779.286481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.286486:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 1995 00000100:00080000:3.0:1788573779.286490:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.286494:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573779.286498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573779.286507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495552000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573779.286516:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573779.286523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.286528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573779.286531:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.286538:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 1995 00000100:00080000:3.0:1788573779.286542:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.286547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573779.286552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573779.286564:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff93a25064d180 x1875453495551744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573779.286568:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495552128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573779.286609:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495551744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573779.286617:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573779.286621:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573779.286624:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495551744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573779.286645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495551872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573779.286662:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573779.286670:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff93a25064f100 x1875453495551872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573779.286678:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495551872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573779.286684:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573779.286685:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573779.286687:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495551872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573779.286691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552000:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573779.286702:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573779.286708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff93a25064ce00 x1875453495552000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573779.286715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495552000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573779.286729:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573779.286731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573779.286733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552000:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573779.286737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495552128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573779.286750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573779.286756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff93a25064e300 x1875453495552128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573779.286763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495552128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573779.286768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573779.286770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573779.286772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495552128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573780.246326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573780.246387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573781.270314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573781.270358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573782.294310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573782.294376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573783.318357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573783.318486:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573784.342322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573784.342435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573784.406381:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573784.406392:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573784.406396:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573784.406407:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2000 00000100:00080000:0.0:1788573784.406417:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573784.406425:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573784.406463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495552256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573784.406490:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573784.406494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573784.406496:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573784.406501:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2000 00000100:00080000:0.0:1788573784.406504:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573784.406508:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573784.406519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573784.406549:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495552384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573784.406564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573784.406564:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495552256:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573784.406585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573784.406588:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573784.406592:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2000 00000100:00080000:0.0:1788573784.406595:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573784.406599:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573784.406603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573784.406614:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495552512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573784.406622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573784.406625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573784.406627:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573784.406630:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2000 00000100:00080000:0.0:1788573784.406635:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573784.406638:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573784.406641:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573784.406648:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495552640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573784.406665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573784.406723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff93a2460d3480 x1875453495552256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.406741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495552256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.406762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.406766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573784.406770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495552256:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573784.406804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573784.406829:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573784.406839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff93a2460d0380 x1875453495552384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.406848:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495552384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.406853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.406856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573784.406858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573784.406862:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552512:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573784.406876:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573784.406882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff93a2460d2680 x1875453495552512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.406891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495552512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.406900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.406904:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573784.406907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552512:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573784.406924:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495552640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573784.406949:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573784.406963:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff93a2460d1500 x1875453495552640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.406976:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495552640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.406985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.406989:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573784.406992:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495552640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573785.366384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573785.366436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573786.390305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573786.390451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573787.414383:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573787.414441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573788.438323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573788.438377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573789.462367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573789.462389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573789.526329:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.526337:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573789.526341:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.526349:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2005 00000100:00080000:3.0:1788573789.526373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.526379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573789.526408:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495552768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573789.526432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.526434:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573789.526436:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.526439:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2005 00000100:00080000:3.0:1788573789.526441:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.526444:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573789.526453:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573789.526462:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495552896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573789.526470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.526473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573789.526474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.526480:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2005 00000100:00100000:2.0:1788573789.526482:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495552768:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573789.526483:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.526489:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573789.526495:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573789.526504:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495553024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573789.526512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.526519:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573789.526521:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.526524:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2005 00000100:00080000:3.0:1788573789.526535:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.526538:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573789.526542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573789.526544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573789.526554:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495553152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573789.526588:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff93a25064d500 x1875453495552768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.526605:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495552768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.526613:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.526616:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573789.526619:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495552768:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573789.526632:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573789.526650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573789.526658:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff93a25064c700 x1875453495552896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.526667:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495552896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.526673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.526675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573789.526678:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495552896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573789.526696:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553024:192.168.202.102@tcp:8:kworker.0 00000800:00000100:2.0:1788573789.526716:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573789.526728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff93a25064ca80 x1875453495553024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.526741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495553024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.526749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.526753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573789.526757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553024:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788573789.526764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495553152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:2.0:1788573789.526787:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573789.526800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff93a25064c380 x1875453495553152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.526822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495553152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.526830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.526834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788573789.526837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495553152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573790.486384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573790.486444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573791.510318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573791.510379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573792.534309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573792.534373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573793.558315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573793.558387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573794.582329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573794.582641:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573794.646336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573794.646345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573794.646348:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573794.646356:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2011 00000100:00080000:2.0:1788573794.646365:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573794.646371:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573794.646403:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495553280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573794.646428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573794.646432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573794.646433:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573794.646436:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2011 00000100:00080000:2.0:1788573794.646438:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573794.646455:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573794.646464:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573794.646473:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495553408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573794.646483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573794.646486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573794.646487:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573794.646492:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495553280:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573794.646493:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2011 00000100:00080000:2.0:1788573794.646501:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573794.646505:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573794.646508:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573794.646517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495553536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573794.646525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573794.646528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573794.646530:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573794.646532:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2011 00000100:00080000:2.0:1788573794.646535:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573794.646537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573794.646540:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573794.646549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495553664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573794.646586:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573794.646612:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553536:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573794.646634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495553664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573795.606381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573795.606658:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573796.630521:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573796.630651:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573797.654552:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573797.654739:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:0.0:1788573797.718477:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788573797.718514:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573797.718528:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573797.718558:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573797.718564:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573797.718571:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788573797.718699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff93a249408e00 x1875453495553408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.718728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495553408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.718742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.718748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573797.718753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573797.718761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff93a24940b800 x1875453495553536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.718772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495553536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.718780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.718797:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573797.718801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553536:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573797.718809:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff93a249409c00 x1875453495553664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.718822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495553664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.718830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.718834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573797.718925:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495553664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573797.718945:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff93a24940a680 x1875453495553280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.718977:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495553280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.718989:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.718994:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573797.718998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495553280:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573798.678394:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573798.678608:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573799.702321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573799.702341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573799.766336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573799.766344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573799.766348:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573799.766359:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2016 00000100:00080000:2.0:1788573799.766370:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573799.766379:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573799.766417:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495553792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573799.766447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573799.766451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573799.766453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573799.766458:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2016 00000100:00080000:2.0:1788573799.766463:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573799.766468:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573799.766479:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573799.766495:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495553920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573799.766501:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495553792:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573799.766519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573799.766525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573799.766533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573799.766539:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2016 00000100:00080000:2.0:1788573799.766544:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573799.766549:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573799.766556:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573799.766570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573799.766591:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495554048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573799.766602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573799.766606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573799.766609:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573799.766613:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2016 00000100:00100000:1.0:1788573799.766617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff93a24940ad80 x1875453495553792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573799.766620:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573799.766630:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573799.766636:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573799.766653:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495554176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573799.766664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495553792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.766677:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.766682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573799.766688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495553792:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573799.766729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573799.766755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573799.766770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff93a249408000 x1875453495553920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.766783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495553920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.766792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.766796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573799.766799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495553920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573799.766806:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554048:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573799.766827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573799.766839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff93a249409180 x1875453495554048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.766860:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495554048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.766882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.766886:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573799.766889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554048:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573799.766896:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495554176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573799.766920:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573799.766932:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff93a249408a80 x1875453495554176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.766945:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495554176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.766954:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.766958:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573799.766962:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495554176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573800.726318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573800.726347:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573801.750340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573801.750367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573802.774371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573802.774399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573803.798338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573803.798365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573804.822358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573804.822410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573804.886317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573804.886326:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573804.886329:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573804.886337:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2021 00000100:00080000:0.0:1788573804.886346:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573804.886352:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573804.886392:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495554304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573804.886415:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573804.886418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573804.886419:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573804.886422:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2021 00000100:00080000:0.0:1788573804.886425:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573804.886427:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573804.886446:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573804.886465:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495554432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573804.886465:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495554304:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573804.886481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573804.886483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573804.886485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573804.886488:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2021 00000100:00080000:0.0:1788573804.886491:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573804.886494:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573804.886498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573804.886507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495554560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573804.886515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573804.886517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573804.886519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573804.886521:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2021 00000800:00000100:1.0:1788573804.886526:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573804.886528:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573804.886533:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573804.886542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573804.886556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495554688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573804.886564:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff93a2460d0000 x1875453495554304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.886583:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495554304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.886601:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.886605:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573804.886607:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495554304:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573804.886627:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573804.886651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573804.886665:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff93a2460d3b80 x1875453495554432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.886678:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495554432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.886688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.886692:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573804.886695:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573804.886699:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554560:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573804.886716:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573804.886727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff93a2460d0700 x1875453495554560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.886739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495554560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.886747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.886750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573804.886754:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554560:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573804.886770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495554688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573804.886793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573804.886803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff93a2460d0a80 x1875453495554688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.886812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495554688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.886817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.886820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573804.886822:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495554688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573805.846295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573805.846372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573806.870353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573806.870431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573807.894356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573807.894451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573808.918367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573808.918448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573809.942295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573809.942451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573810.006344:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573810.006355:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573810.006360:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573810.006372:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2026 00000100:00080000:0.0:1788573810.006383:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573810.006390:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573810.006436:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495554816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573810.006464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573810.006467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573810.006469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573810.006476:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2026 00000100:00080000:0.0:1788573810.006479:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573810.006483:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573810.006494:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573810.006504:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495554944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573810.006512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573810.006517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573810.006519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573810.006519:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495554816:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573810.006528:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2026 00000100:00080000:0.0:1788573810.006531:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573810.006533:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573810.006556:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573810.006574:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495555072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573810.006583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573810.006586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573810.006587:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573810.006590:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2026 00000100:00080000:0.0:1788573810.006594:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573810.006598:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573810.006600:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573810.006606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573810.006614:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495555200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573810.006647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff93a2460d0a80 x1875453495554816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.006664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495554816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.006673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.006676:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573810.006679:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495554816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573810.006701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573810.006717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573810.006725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff93a2460d0700 x1875453495554944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.006741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495554944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.006748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.006750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573810.006752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495554944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573810.006756:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555072:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573810.006770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573810.006777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff93a2460d3b80 x1875453495555072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.006798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495555072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.006804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.006807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573810.006809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555072:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573810.006812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495555200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573810.006825:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573810.006832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff93a2460d0000 x1875453495555200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.006839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495555200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.006854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.006856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573810.006858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495555200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573810.966357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573810.966438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573811.990353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573811.990420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573813.014372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573813.014448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573814.038311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573814.038437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573815.062404:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573815.062479:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573815.126420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573815.126435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573815.126440:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573815.126453:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2031 00000100:00080000:2.0:1788573815.126466:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573815.126474:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573815.126520:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495555328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573815.126549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573815.126552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573815.126554:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573815.126557:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2031 00000100:00080000:2.0:1788573815.126559:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573815.126583:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573815.126593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573815.126605:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495555456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573815.126613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573815.126616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573815.126618:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573815.126621:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2031 00000100:00080000:2.0:1788573815.126623:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573815.126626:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573815.126630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573815.126643:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495555584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573815.126645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495555328:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573815.126664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573815.126667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573815.126668:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573815.126671:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2031 00000100:00080000:2.0:1788573815.126672:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573815.126674:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573815.126678:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573815.126688:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495555712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573815.126736:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573815.126795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff93a24940aa00 x1875453495555328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.126831:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495555328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.126843:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.126848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573815.126852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495555328:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573815.126861:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573815.126884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573815.126897:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff93a249408700 x1875453495555456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.126908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495555456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.126917:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.126920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573815.126924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573815.126961:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555584:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573815.126982:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573815.126993:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff93a249409880 x1875453495555584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.127006:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495555584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.127013:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.127017:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573815.127021:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555584:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573815.127027:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495555712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573815.127046:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573815.127057:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff93a249409500 x1875453495555712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.127069:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495555712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.127078:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.127081:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573815.127085:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495555712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573816.086312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573816.086378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573817.110347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573817.110404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573818.134314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573818.134438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573819.158334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573819.158393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573820.182284:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573820.182364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573820.246338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573820.246361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573820.246364:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573820.246373:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2036 00000100:00080000:2.0:1788573820.246382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573820.246388:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573820.246418:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495555840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573820.246446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573820.246449:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573820.246451:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573820.246454:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2036 00000100:00080000:2.0:1788573820.246456:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573820.246459:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573820.246467:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573820.246479:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495555968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573820.246490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573820.246495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495555840:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573820.246496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573820.246501:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573820.246519:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2036 00000100:00080000:2.0:1788573820.246528:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573820.246533:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573820.246538:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573820.246549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495556096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573820.246566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573820.246568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573820.246573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573820.246576:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573820.246582:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2036 00000100:00080000:2.0:1788573820.246588:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573820.246592:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573820.246599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573820.246613:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495556224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573820.246625:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff93a24940b480 x1875453495555840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573820.246650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495555840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573820.246662:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573820.246666:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573820.246671:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495555840:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573820.246692:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573820.246712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573820.246738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff93a24940bb80 x1875453495555968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573820.246750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495555968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573820.246774:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573820.246777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573820.246779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495555968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573820.246792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556096:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573820.246810:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573820.246818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff93a24940b100 x1875453495556096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573820.246826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495556096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573820.246833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573820.246837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573820.246840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556096:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573820.246844:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495556224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573820.246858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573820.246865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff93a249409500 x1875453495556224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573820.246872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495556224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573820.246877:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573820.246879:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573820.246881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495556224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573821.206300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573821.206368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573822.230371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573822.230454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573823.254354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573823.254429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573824.278338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573824.278408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573825.302320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573825.302406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573825.366393:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573825.366409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573825.366414:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573825.366430:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2041 00000100:00080000:3.0:1788573825.366445:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573825.366455:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573825.366507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495556352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573825.366541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573825.366546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573825.366548:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573825.366553:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2041 00000100:00080000:3.0:1788573825.366557:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573825.366562:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573825.366576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573825.366594:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495556480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573825.366605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573825.366610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573825.366612:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573825.366618:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2041 00000100:00080000:3.0:1788573825.366622:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573825.366640:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573825.366650:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495556352:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788573825.366668:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573825.366685:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495556608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573825.366700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573825.366704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573825.366706:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573825.366711:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2041 00000100:00080000:3.0:1788573825.366716:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573825.366721:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573825.366726:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573825.366727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573825.366745:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495556736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573825.366794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff93a25064df80 x1875453495556352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.366821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495556352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.366831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.366836:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573825.366840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495556352:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573825.366851:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573825.366878:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573825.366893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff93a25064c380 x1875453495556480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.366908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495556480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.366917:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.366920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573825.366922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573825.366955:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556608:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573825.366980:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573825.366993:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff93a25064ca80 x1875453495556608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.367006:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495556608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.367022:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.367025:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573825.367027:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556608:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573825.367032:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495556736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573825.367048:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573825.367056:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff93a25064c700 x1875453495556736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.367063:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495556736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.367069:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.367071:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573825.367073:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495556736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573826.326374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573826.326472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573827.350339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573827.350399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573828.374332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573828.374407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573829.398380:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573829.398458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573830.422365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573830.422434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573830.486346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573830.486358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573830.486364:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573830.486375:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2046 00000100:00080000:0.0:1788573830.486387:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573830.486395:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573830.486438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495556864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573830.486464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573830.486467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573830.486469:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573830.486474:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2046 00000100:00080000:0.0:1788573830.486478:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573830.486482:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573830.486495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573830.486515:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495556992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573830.486516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495556864:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573830.486534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573830.486561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573830.486563:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573830.486567:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2046 00000100:00080000:0.0:1788573830.486570:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573830.486591:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573830.486597:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573830.486607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495557120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573830.486615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573830.486621:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1788573830.486622:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573830.486623:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573830.486630:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2046 00000100:00080000:0.0:1788573830.486633:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573830.486636:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573830.486640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573830.486649:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495557248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573830.486682:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff93a2460d1500 x1875453495556864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.486706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495556864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.486718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.486724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573830.486728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495556864:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573830.486775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573830.486803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573830.486818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff93a2460d2680 x1875453495556992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.486832:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495556992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.486840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.486844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573830.486848:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495556992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573830.486854:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495557120:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573830.486876:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573830.486888:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff93a2460d0380 x1875453495557120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.486900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495557120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.486908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.486912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573830.486916:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495557120:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573830.486922:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495557248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573830.486953:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573830.486969:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff93a2460d3480 x1875453495557248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.486989:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495557248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.487005:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.487008:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573830.487012:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495557248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573831.446318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573831.446385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573832.470330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573832.470422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573833.494298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573833.494344:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573834.518379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573834.518450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573835.542302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573835.542419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573835.606358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573835.606366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573835.606370:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573835.606384:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2052 00000100:00080000:2.0:1788573835.606395:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573835.606402:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573835.606440:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495557376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573835.606476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573835.606491:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573835.606493:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573835.606496:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2052 00000100:00080000:2.0:1788573835.606500:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573835.606505:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573835.606514:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573835.606526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495557504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573835.606528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495557376:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573835.606543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573835.606546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573835.606548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573835.606551:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2052 00000100:00080000:2.0:1788573835.606554:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573835.606556:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573835.606560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573835.606568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495557632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573835.606575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573835.606578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573835.606579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573835.606582:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2052 00000100:00080000:2.0:1788573835.606584:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573835.606586:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788573835.606594:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:2.0:1788573835.606597:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573835.606608:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495557760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573835.606643:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff93a249409880 x1875453495557376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.606672:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495557376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.606681:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.606684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573835.606687:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495557376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573835.606706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495557504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573835.606724:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573835.606734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff93a249408700 x1875453495557504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.606742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495557504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.606759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.606762:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573835.606764:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495557504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573835.606768:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495557632:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573835.606783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573835.606790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff93a24940aa00 x1875453495557632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.606797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495557632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.606802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.606804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573835.606806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495557632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573835.606810:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495557760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573835.606863:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573835.606887:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff93a249408a80 x1875453495557760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.606896:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495557760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.606901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.606903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573835.606911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495557760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573836.566336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573836.566391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573837.590381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573837.590454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573838.614362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573838.614439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573839.638375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573839.638469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573840.662338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573840.662448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573840.726415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.726424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573840.726428:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.726438:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2057 00000100:00080000:0.0:1788573840.726448:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.726455:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573840.726489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495557888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573840.726515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.726518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573840.726532:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.726536:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2057 00000100:00080000:0.0:1788573840.726538:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.726541:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573840.726549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573840.726561:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495558016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573840.726570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.726573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573840.726574:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.726586:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2057 00000100:00100000:1.0:1788573840.726589:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495557888:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573840.726590:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.726601:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573840.726608:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573840.726616:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495558144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573840.726623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.726626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573840.726628:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.726631:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2057 00000100:00080000:0.0:1788573840.726633:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.726635:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573840.726638:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573840.726653:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495558272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573840.726681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573840.726728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff93a2460d2a00 x1875453495557888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.726744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495557888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.726754:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.726758:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573840.726761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495557888:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573840.726798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573840.726819:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573840.726833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff93a2460d1180 x1875453495558016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.726846:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495558016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.726854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.726857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573840.726861:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573840.726867:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558144:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573840.726890:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573840.726919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff93a2460d2300 x1875453495558144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.726931:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495558144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.726939:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.726942:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573840.726946:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558144:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573840.726951:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495558272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573840.726974:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573840.726987:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff93a2460d1f80 x1875453495558272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.726999:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495558272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.727008:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.727012:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573840.727015:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495558272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573841.686333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573841.686423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573842.710307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573842.710400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573843.734347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573843.734407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573844.758326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573844.758372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573845.782366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573845.782439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573845.846351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573845.846362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573845.846366:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573845.846377:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2062 00000100:00080000:1.0:1788573845.846387:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573845.846393:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573845.846443:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495558400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573845.846474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573845.846477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573845.846479:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573845.846482:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2062 00000100:00080000:1.0:1788573845.846486:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573845.846489:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573845.846498:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573845.846509:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495558528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573845.846516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495558400:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573845.846520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573845.846524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573845.846525:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573845.846529:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2062 00000100:00080000:1.0:1788573845.846531:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573845.846534:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573845.846537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573845.846545:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495558656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573845.846551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573845.846554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573845.846556:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573845.846565:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2062 00000100:00080000:1.0:1788573845.846569:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573845.846571:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788573845.846575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788573845.846579:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573845.846589:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495558784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573845.846620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff93a258da1c00 x1875453495558400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.846637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495558400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.846644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.846648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573845.846650:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495558400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573845.846670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573845.846685:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573845.846693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff93a258da0700 x1875453495558528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.846701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495558528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.846707:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.846710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573845.846712:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573845.846724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558656:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573845.846738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573845.846745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff93a258da0a80 x1875453495558656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.846753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495558656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.846758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.846760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573845.846762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495558656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573845.846765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495558784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573845.846775:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573845.846781:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff93a258da2680 x1875453495558784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.846791:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495558784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.846796:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.846798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573845.846800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495558784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573846.806374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573846.806435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573847.830335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573847.830418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573848.854330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573848.854401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573849.878315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573849.878357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573850.902331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573850.902397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573850.966362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573850.966374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573850.966378:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573850.966397:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2067 00000100:00080000:2.0:1788573850.966408:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573850.966427:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573850.966473:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495558912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573850.966514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573850.966518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573850.966544:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573850.966550:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2067 00000100:00080000:2.0:1788573850.966555:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573850.966567:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495558912:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573850.966575:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573850.966588:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573850.966615:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495559040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573850.966627:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573850.966642:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573850.966647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573850.966650:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573850.966656:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2067 00000100:00080000:2.0:1788573850.966662:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573850.966667:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573850.966690:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573850.966699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff93a249409180 x1875453495558912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573850.966720:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495559168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573850.966723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495558912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573850.966747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573850.966751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:2.0:1788573850.966758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573850.966759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495558912:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573850.966772:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573850.966776:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573850.966782:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2067 00000100:00080000:2.0:1788573850.966786:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573850.966791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559040:192.168.202.102@tcp:38:kworker.0 00000100:00080000:2.0:1788573850.966796:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573850.966805:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573850.966826:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573850.966830:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495559296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573850.966845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff93a249408000 x1875453495559040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573850.966861:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495559040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573850.966870:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573850.966874:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573850.966878:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573850.966884:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559168:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573850.966904:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573850.966914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff93a24940ad80 x1875453495559168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573850.966935:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495559168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573850.966944:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573850.966947:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573850.966951:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559168:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573850.966964:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495559296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573850.966983:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573850.966994:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff93a24940a680 x1875453495559296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573850.967005:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495559296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573850.967013:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573850.967017:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573850.967020:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495559296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573851.926384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573851.926451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573852.950332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573852.950385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573853.974378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573853.974450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573854.998352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573854.998412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573856.022329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573856.022400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573856.086376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573856.086388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573856.086393:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573856.086405:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2072 00000100:00080000:2.0:1788573856.086417:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573856.086423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573856.086470:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495559424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573856.086499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573856.086502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573856.086504:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573856.086508:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2072 00000100:00080000:2.0:1788573856.086510:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573856.086513:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573856.086522:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573856.086531:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495559552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573856.086540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573856.086568:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573856.086571:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573856.086575:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2072 00000100:00080000:2.0:1788573856.086578:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573856.086584:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573856.086594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573856.086600:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495559424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573856.086605:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495559680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573856.086623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573856.086628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573856.086629:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573856.086635:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2072 00000100:00080000:2.0:1788573856.086638:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573856.086641:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573856.086645:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573856.086654:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495559808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573856.086676:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573856.086734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff93a249409c00 x1875453495559424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573856.086767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495559424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573856.086776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573856.086780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573856.086782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495559424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573856.086789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573856.086813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573856.086827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff93a24940b800 x1875453495559552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573856.086840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495559552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573856.086846:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573856.086849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573856.086851:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573856.086871:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559680:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573856.086888:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573856.086895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff93a249408e00 x1875453495559680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573856.086903:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495559680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573856.086917:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573856.086920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573856.086922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495559680:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573856.086927:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495559808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573856.086941:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573856.086948:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff93a24940a680 x1875453495559808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573856.086955:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495559808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573856.086960:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573856.086962:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573856.086964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495559808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573857.046334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573857.046392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573858.070320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573858.070378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573859.094327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573859.094385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573860.118323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573860.118386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573861.142782:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573861.142831:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573861.206315:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573861.206324:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573861.206327:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573861.206337:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2077 00000100:00080000:2.0:1788573861.206346:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573861.206353:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573861.206391:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495559936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573861.206433:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573861.206436:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573861.206437:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573861.206441:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2077 00000100:00080000:2.0:1788573861.206444:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573861.206447:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573861.206463:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573861.206476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495560064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573861.206484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573861.206486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573861.206488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573861.206491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2077 00000100:00080000:2.0:1788573861.206494:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573861.206497:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573861.206500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573861.206506:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495560192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573861.206515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573861.206521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573861.206524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573861.206526:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495559936:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573861.206528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2077 00000100:00080000:2.0:1788573861.206540:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573861.206545:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573861.206549:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573861.206560:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495560320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573861.206683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573861.206737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573861.206771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495560320:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788573862.167118:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573862.167520:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573863.190697:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573863.191019:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573864.214364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573864.214437:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:1.0:1788573864.278436:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788573864.278473:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573864.278487:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573864.278516:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573864.278522:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788573864.278528:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573864.278575:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff93a249409180 x1875453495560192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573864.278596:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495560192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573864.278605:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573864.278609:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573864.278612:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573864.278617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff93a249408a80 x1875453495560320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573864.278625:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495560320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573864.278630:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573864.278633:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573864.278635:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495560320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573864.278639:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff93a24940ad80 x1875453495559936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573864.278646:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495559936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573864.278651:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573864.278653:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573864.278655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495559936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573864.278672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff93a249408000 x1875453495560064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573864.278686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495560064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573864.278694:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573864.278698:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573864.278701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560064:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788573865.238380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573865.238812:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573866.262338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788573866.262392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573866.326328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573866.326338:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573866.326341:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573866.326350:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2082 00000100:00080000:2.0:1788573866.326358:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573866.326365:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573866.326400:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495560448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573866.326421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573866.326424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573866.326426:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573866.326429:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2082 00000100:00080000:2.0:1788573866.326431:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573866.326434:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573866.326442:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573866.326452:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495560576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573866.326476:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495560448:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573866.326495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573866.326498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573866.326501:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573866.326507:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2082 00000100:00080000:2.0:1788573866.326512:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573866.326516:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573866.326523:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573866.326539:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495560704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573866.326551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573866.326557:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573866.326559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573866.326563:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573866.326568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2082 00000100:00080000:2.0:1788573866.326574:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573866.326579:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573866.326584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573866.326597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495560832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573866.326622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff93a24940aa00 x1875453495560448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573866.326647:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495560448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573866.326658:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573866.326662:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573866.326667:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495560448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573866.326691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573866.326715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573866.326728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff93a249408700 x1875453495560576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573866.326753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495560576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573866.326761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573866.326765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573866.326768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573866.326775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560704:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573866.326797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573866.326810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff93a249409880 x1875453495560704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573866.326822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495560704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573866.326831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573866.326834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573866.326838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495560704:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573866.326845:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495560832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573866.326880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573866.326892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff93a249409500 x1875453495560832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573866.326905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495560832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573866.326925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573866.326929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573866.326932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495560832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573867.286331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573867.286409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573868.310307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573868.310371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573869.334347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573869.334421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573870.358399:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573870.358449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573871.382340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573871.382371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573871.446367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.446382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573871.446388:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.446404:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2087 00000100:00080000:3.0:1788573871.446418:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.446429:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573871.446486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495560960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.446526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.446530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.446533:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.446538:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2087 00000100:00080000:3.0:1788573871.446542:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.446547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573871.446560:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.446575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495561088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573871.446579:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495560960:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573871.446601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.446604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.446606:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.446611:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2087 00000100:00080000:3.0:1788573871.446614:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.446617:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573871.446622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.446631:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495561216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.446639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.446642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.446644:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.446646:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2087 00000100:00080000:3.0:1788573871.446648:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.446651:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573871.446654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.446660:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495561344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573871.446679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573871.446726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff93a25064d500 x1875453495560960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.446744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495560960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.446779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.446782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573871.446786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495560960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573871.446815:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573871.446852:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573871.446864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff93a25064e300 x1875453495561088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.446874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495561088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.446880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.446883:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573871.446885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573871.446890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561216:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573871.446906:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573871.446913:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff93a25064ce00 x1875453495561216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.446921:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495561216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.446927:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.446928:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573871.446930:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573871.446940:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495561344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573871.446955:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573871.446962:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff93a25064f100 x1875453495561344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.446970:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495561344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.446975:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.446977:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573871.446979:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495561344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573872.406348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573872.406386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573873.430306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573873.430374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573874.454338:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573874.454424:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573875.478339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573875.478399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573876.502301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573876.502358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573876.566328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.566338:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573876.566342:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.566350:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2093 00000100:00080000:2.0:1788573876.566374:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.566381:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573876.566412:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495561472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.566436:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.566438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573876.566440:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.566443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2093 00000100:00080000:2.0:1788573876.566445:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.566447:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573876.566455:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573876.566463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495561600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.566470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.566472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573876.566474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.566477:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2093 00000100:00080000:2.0:1788573876.566478:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.566481:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573876.566484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573876.566491:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495561728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.566499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573876.566504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495561472:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573876.566510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573876.566513:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.566517:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2093 00000100:00080000:2.0:1788573876.566520:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.566523:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573876.566528:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573876.566538:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495561856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573876.566566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573876.566619:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff93a24940b100 x1875453495561472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573876.566638:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495561472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573876.566673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573876.566677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573876.566681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495561472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573876.566693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573876.566712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573876.566722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff93a24940bb80 x1875453495561600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573876.566731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495561600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573876.566736:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573876.566739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573876.566741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573876.566760:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561728:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573876.566773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573876.566779:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff93a24940b480 x1875453495561728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573876.566786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495561728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573876.566792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573876.566794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573876.566796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495561728:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573876.566799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495561856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573876.566811:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573876.566821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff93a249409500 x1875453495561856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573876.566842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495561856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573876.566850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573876.566853:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573876.566855:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495561856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573877.526366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573877.526397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573878.550378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573878.550407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573879.574325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573879.574355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573880.598332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573880.598376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573881.622327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573881.622345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573881.686402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573881.686414:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573881.686418:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573881.686429:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2098 00000100:00080000:0.0:1788573881.686439:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573881.686448:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573881.686491:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495561984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573881.686519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573881.686522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573881.686523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573881.686526:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2098 00000100:00080000:0.0:1788573881.686529:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573881.686532:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573881.686540:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573881.686564:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495562112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573881.686580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495561984:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573881.686597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573881.686601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573881.686603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573881.686610:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2098 00000100:00080000:0.0:1788573881.686615:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573881.686620:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573881.686626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573881.686639:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573881.686643:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495562240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573881.686653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573881.686656:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573881.686658:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573881.686661:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2098 00000100:00080000:0.0:1788573881.686664:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573881.686667:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573881.686671:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573881.686681:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff93a2460d1f80 x1875453495561984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573881.686685:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495562368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573881.686729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495561984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.686739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.686743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573881.686747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495561984:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573881.686765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573881.686784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573881.686794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff93a2460d2300 x1875453495562112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.686810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495562112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.686816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.686818:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573881.686820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573881.686824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562240:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573881.686838:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573881.686845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff93a2460d1180 x1875453495562240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.686852:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495562240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.686857:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.686859:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573881.686861:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562240:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573881.686864:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495562368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573881.686874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573881.686880:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff93a2460d2a00 x1875453495562368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.686887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495562368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.686901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.686903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573881.686905:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495562368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573882.646311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573882.646347:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573883.670588:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573883.670623:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573884.694329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573884.694342:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573885.718384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573885.718386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573886.742376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573886.742409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573886.806289:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573886.806297:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573886.806300:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573886.806309:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2103 00000100:00080000:0.0:1788573886.806334:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573886.806339:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573886.806368:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495562496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573886.806393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573886.806396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573886.806397:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573886.806401:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2103 00000100:00080000:0.0:1788573886.806403:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573886.806405:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573886.806416:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573886.806425:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495562624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573886.806433:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573886.806436:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573886.806451:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573886.806457:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2103 00000100:00080000:0.0:1788573886.806461:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573886.806465:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573886.806470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573886.806480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495562752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573886.806489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573886.806496:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495562496:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573886.806497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573886.806501:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573886.806518:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2103 00000100:00080000:0.0:1788573886.806524:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573886.806530:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573886.806537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573886.806554:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495562880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573886.806590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573886.806640:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff93a2460d3480 x1875453495562496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.806658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495562496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.806667:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.806671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573886.806675:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495562496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573886.806680:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573886.806696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573886.806704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff93a2460d0380 x1875453495562624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.806713:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495562624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.806730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.806733:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573886.806735:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573886.806762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562752:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573886.806779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573886.806786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff93a2460d2680 x1875453495562752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.806796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495562752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.806805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.806809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573886.806812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495562752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573886.806819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495562880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573886.806841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573886.806854:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff93a2460d1500 x1875453495562880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.806863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495562880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.806869:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.806871:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573886.806879:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495562880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573887.766327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573887.766348:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573888.790378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573888.790432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573889.814368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573889.814412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573890.838336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573890.838369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573891.862736:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573891.862752:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573891.926331:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573891.926355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573891.926359:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573891.926370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2108 00000100:00080000:0.0:1788573891.926381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573891.926388:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573891.926425:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495563008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573891.926455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573891.926459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573891.926461:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573891.926466:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2108 00000100:00080000:0.0:1788573891.926470:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573891.926475:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573891.926486:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573891.926499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495563008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573891.926505:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495563136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573891.926536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573891.926540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573891.926542:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573891.926547:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2108 00000100:00080000:0.0:1788573891.926553:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573891.926554:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573891.926563:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573891.926571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573891.926588:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495563264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573891.926596:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff93a2460d0000 x1875453495563008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573891.926606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573891.926612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573891.926614:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573891.926615:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495563008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573891.926623:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2108 00000100:00080000:1.0:1788573891.926629:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573891.926631:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573891.926634:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:0.0:1788573891.926640:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573891.926641:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495563008:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573891.926652:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573891.926666:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788573891.926671:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495563392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573891.926687:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573891.926699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff93a2460d3b80 x1875453495563136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573891.926709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495563136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573891.926714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573891.926717:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573891.926725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573891.926730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563264:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573891.926754:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573891.926762:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff93a2460d0700 x1875453495563264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573891.926770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495563264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573891.926775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573891.926777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573891.926779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563264:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573891.926790:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495563392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573891.926801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573891.926807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff93a2460d0a80 x1875453495563392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573891.926814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495563392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573891.926819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573891.926821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573891.926823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495563392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573892.886349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573892.886365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573893.910318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573893.910343:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573894.934922:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573894.935122:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788573895.958738:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573895.958855:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573896.982325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573896.982376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573897.046354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573897.046362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573897.046366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573897.046375:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2113 00000100:00080000:2.0:1788573897.046384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573897.046389:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573897.046422:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495563520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573897.046443:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573897.046446:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573897.046447:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573897.046451:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2113 00000100:00080000:2.0:1788573897.046453:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573897.046455:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573897.046479:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573897.046488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495563520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573897.046492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495563648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573897.046502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573897.046504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573897.046505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573897.046508:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2113 00000100:00080000:2.0:1788573897.046511:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573897.046513:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573897.046517:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573897.046525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495563776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573897.046532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573897.046535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573897.046536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573897.046539:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2113 00000100:00080000:2.0:1788573897.046541:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573897.046543:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573897.046547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573897.046551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573897.046559:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495563904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573897.046581:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff93a249409880 x1875453495563520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573897.046595:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495563520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573897.046603:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573897.046606:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573897.046609:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495563520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573897.046628:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573897.046642:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573897.046650:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff93a249408700 x1875453495563648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573897.046657:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495563648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573897.046663:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573897.046665:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573897.046667:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573897.046670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563776:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573897.046681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573897.046697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff93a24940aa00 x1875453495563776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573897.046704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495563776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573897.046710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573897.046712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573897.046714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495563776:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573897.046718:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495563904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573897.046729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573897.046735:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff93a249408000 x1875453495563904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573897.046742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495563904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573897.046746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573897.046748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573897.046750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495563904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573898.006306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573898.006367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573899.031300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573899.031398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573900.054794:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573900.054910:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573901.078321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573901.078416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573902.102360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573902.102458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573902.166391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573902.166409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573902.166414:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573902.166426:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2118 00000100:00080000:0.0:1788573902.166439:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573902.166455:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573902.166502:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495564032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573902.166534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573902.166537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573902.166539:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573902.166542:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2118 00000100:00080000:0.0:1788573902.166545:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573902.166548:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573902.166558:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573902.166569:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495564160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573902.166577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573902.166580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573902.166582:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573902.166585:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2118 00000100:00080000:0.0:1788573902.166587:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573902.166598:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573902.166600:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495564032:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788573902.166625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573902.166642:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495564288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573902.166668:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573902.166672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573902.166675:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573902.166680:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573902.166684:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2118 00000100:00080000:0.0:1788573902.166692:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573902.166698:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573902.166705:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573902.166728:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495564416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573902.166743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff93a2460d0a80 x1875453495564032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573902.166769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495564032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573902.166790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573902.166796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573902.166801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495564032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573902.166808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573902.166835:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573902.166850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff93a2460d0700 x1875453495564160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573902.166864:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495564160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573902.166872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573902.166876:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573902.166879:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573902.166911:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564288:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573902.166932:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573902.166944:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff93a2460d3b80 x1875453495564288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573902.166957:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495564288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573902.166966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573902.166970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573902.166974:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573902.166989:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495564416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573902.167011:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573902.167024:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff93a2460d0000 x1875453495564416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573902.167036:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495564416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573902.167045:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573902.167048:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573902.167051:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495564416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573903.126319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573903.126395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573904.150364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573904.150502:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573905.174355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573905.174434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573906.198368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573906.198448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573907.222344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573907.222662:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573907.286366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573907.286382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573907.286388:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573907.286402:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2123 00000100:00080000:2.0:1788573907.286417:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573907.286426:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573907.286480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495564544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573907.286515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573907.286520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573907.286523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573907.286528:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2123 00000100:00080000:2.0:1788573907.286533:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573907.286538:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573907.286551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573907.286568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495564672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573907.286579:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495564544:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573907.286584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573907.286589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573907.286612:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573907.286619:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2123 00000100:00080000:2.0:1788573907.286624:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573907.286629:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573907.286636:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573907.286651:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495564800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573907.286664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573907.286665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573907.286672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573907.286675:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573907.286681:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2123 00000100:00080000:2.0:1788573907.286686:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573907.286692:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573907.286697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573907.286712:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495564928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573907.286720:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff93a24940ad80 x1875453495564544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573907.286742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495564544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573907.286751:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573907.286755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573907.286759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495564544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573907.286800:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573907.286826:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573907.286841:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff93a249408a80 x1875453495564672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573907.286858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495564672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573907.286865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573907.286867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573907.286870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573907.286874:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564800:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573907.286891:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573907.286899:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff93a249409180 x1875453495564800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573907.286907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495564800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573907.286912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573907.286914:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573907.286916:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495564800:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573907.286919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495564928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573907.286946:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573907.286953:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff93a24940a680 x1875453495564928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573907.286961:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495564928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573907.286977:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573907.286979:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573907.286983:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495564928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573908.246357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573908.247225:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573909.270332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573909.270403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573910.294394:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573910.294448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573911.318316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573911.318515:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573912.342419:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573912.342606:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573912.406397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.406409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573912.406414:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.406425:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2128 00000100:00080000:3.0:1788573912.406436:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.406444:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573912.406490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495565056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573912.406521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.406526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.406528:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.406533:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2128 00000100:00080000:3.0:1788573912.406552:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.406559:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573912.406570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573912.406595:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495565184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573912.406598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495565056:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573912.406623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.406633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.406637:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.406645:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2128 00000100:00080000:3.0:1788573912.406651:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.406657:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573912.406666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573912.406688:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495565312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573912.406695:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788573912.406712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.406717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.406721:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.406728:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2128 00000100:00080000:3.0:1788573912.406733:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.406739:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573912.406746:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573912.406751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff93a25064d180 x1875453495565056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573912.406765:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495565440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573912.406775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495565056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.406793:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.406805:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573912.406813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495565056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573912.406848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573912.406875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573912.406884:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff93a25064c000 x1875453495565184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.406892:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495565184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.406899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.406901:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573912.406903:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573912.406921:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565312:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573912.406946:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573912.406954:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff93a25064f480 x1875453495565312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.406963:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495565312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.406968:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.406971:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573912.406973:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565312:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573912.406977:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495565440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573912.406988:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573912.406994:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff93a25064fb80 x1875453495565440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.407001:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495565440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.407006:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.407009:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573912.407033:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495565440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573913.366359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573913.366415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573914.390350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573914.390445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573915.414925:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573915.415068:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788573916.438573:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573916.438702:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788573917.462349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573917.462445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573917.526348:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573917.526356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573917.526375:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573917.526383:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2133 00000100:00080000:0.0:1788573917.526392:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573917.526398:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573917.526426:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495565568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573917.526449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573917.526452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573917.526453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573917.526457:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2133 00000100:00080000:0.0:1788573917.526459:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573917.526461:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573917.526474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573917.526484:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495565696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573917.526493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573917.526495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495565568:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573917.526497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573917.526500:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573917.526504:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2133 00000100:00080000:0.0:1788573917.526508:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573917.526513:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573917.526517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573917.526524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495565824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573917.526544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573917.526546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573917.526548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573917.526551:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2133 00000100:00080000:0.0:1788573917.526555:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573917.526558:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573917.526561:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573917.526562:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788573917.526575:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495565952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573917.526609:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff93a2460d1500 x1875453495565568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.526627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495565568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.526636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.526639:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573917.526647:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495565568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573917.526665:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573917.526681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573917.526689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff93a2460d2680 x1875453495565696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.526697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495565696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.526703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.526705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573917.526707:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573917.526711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565824:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573917.526722:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573917.526728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff93a2460d0380 x1875453495565824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.526747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495565824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.526752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.526754:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573917.526756:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495565824:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573917.526760:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495565952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573917.526772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573917.526779:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff93a2460d3480 x1875453495565952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.526786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495565952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.526791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.526792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573917.526794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495565952:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573918.486347:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573918.486352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788573919.510482:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573919.510749:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573920.534358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573920.534528:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573921.558380:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573921.558403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573922.582293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573922.582429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573922.646356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.646368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573922.646373:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.646383:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2139 00000100:00080000:3.0:1788573922.646391:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.646398:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573922.646431:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495566080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.646460:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.646464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573922.646471:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.646475:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2139 00000100:00080000:3.0:1788573922.646479:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.646482:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573922.646490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573922.646501:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495566208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.646509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.646511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573922.646513:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.646516:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2139 00000100:00080000:3.0:1788573922.646518:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573922.646527:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495566080:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573922.646537:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573922.646542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573922.646550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495566336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.646558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.646561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573922.646563:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.646565:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2139 00000100:00080000:3.0:1788573922.646568:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.646571:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573922.646574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573922.646580:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495566464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573922.646597:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573922.646649:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff93a25064e680 x1875453495566080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.646668:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495566080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.646688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.646692:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573922.646696:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495566080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573922.646704:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573922.646727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573922.646740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff93a25064ed80 x1875453495566208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.646755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495566208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.646763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.646767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573922.646770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573922.646795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566336:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788573922.646815:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573922.646825:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff93a25064dc00 x1875453495566336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.646839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495566336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.646848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.646852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573922.646855:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566336:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573922.646868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495566464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788573922.646888:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573922.646900:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff93a25064d880 x1875453495566464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.646913:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495566464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.646921:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.646925:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573922.646929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495566464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788573923.606344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573923.606394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573924.630361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573924.630437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573925.654295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573925.654451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573926.678350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573926.678550:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573927.702553:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573927.702763:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573927.766313:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.766330:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573927.766336:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.766365:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2144 00000100:00080000:3.0:1788573927.766379:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.766388:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573927.766442:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495566592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.766478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.766483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573927.766516:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.766523:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2144 00000100:00080000:3.0:1788573927.766529:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573927.766540:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495566592:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788573927.766551:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573927.766567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573927.766589:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495566720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.766603:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.766608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573927.766610:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.766616:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2144 00000100:00080000:3.0:1788573927.766621:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.766625:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573927.766632:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573927.766647:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495566848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.766660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.766665:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573927.766668:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573927.766676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566720:192.168.202.102@tcp:38:kworker.0 00000100:00080000:3.0:1788573927.766676:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2144 00000100:00080000:3.0:1788573927.766684:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.766691:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788573927.766698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573927.766719:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573927.766727:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495566976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573927.766757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495566976:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788573928.726769:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573928.727698:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573929.750532:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573929.750941:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573930.774354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573930.775290:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788573930.838470:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788573930.838510:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573930.838527:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573930.838571:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573930.838583:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788573930.838595:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788573930.838622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573930] req@ffff93a25064ed80 x1875453495566976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573930.838650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495566976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573930.838663:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573930.838668:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573930.838673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495566976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573930.838703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573930] req@ffff93a25064d880 x1875453495566720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573930.838719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495566720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573930.838729:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573930.838733:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573930.838737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788573930.838744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573930] req@ffff93a25064dc00 x1875453495566848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573930.838769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495566848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573930.838778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573930.838781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573930.838785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495566848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788573930.838792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573930] req@ffff93a25064ea00 x1875453495566592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573930.838803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495566592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573930.838810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573930.838813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788573930.838816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495566592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573931.798366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573931.799874:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573932.822347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573932.822422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573932.886351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573932.886361:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573932.886365:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573932.886377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2149 00000100:00080000:1.0:1788573932.886387:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573932.886396:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573932.886431:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495567104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573932.886469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573932.886473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573932.886475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573932.886480:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2149 00000100:00080000:1.0:1788573932.886483:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573932.886488:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573932.886498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573932.886512:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495567232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573932.886523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573932.886527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573932.886530:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573932.886532:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495567104:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573932.886539:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2149 00000100:00080000:1.0:1788573932.886544:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573932.886548:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573932.886569:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573932.886583:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495567360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573932.886594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573932.886597:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573932.886599:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573932.886604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788573932.886614:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2149 00000100:00080000:1.0:1788573932.886620:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573932.886626:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573932.886631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573932.886645:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495567488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573932.886667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff93a258da1f80 x1875453495567104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573932.886689:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495567104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573932.886700:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573932.886704:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573932.886709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495567104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573932.886739:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573932.886765:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573932.886780:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff93a258da0e00 x1875453495567232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573932.886793:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495567232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573932.886801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573932.886816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573932.886819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573932.886826:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567360:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573932.886850:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573932.886863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff93a258da3800 x1875453495567360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573932.886876:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495567360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573932.886885:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573932.886888:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573932.886892:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567360:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573932.886899:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495567488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573932.886922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573932.886935:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff93a258da2680 x1875453495567488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573932.886946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495567488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573932.886955:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573932.886959:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573932.886962:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495567488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573933.846332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573933.846387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573934.870313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573934.870377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573935.894372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573935.894442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573936.918397:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573936.918467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573937.942339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573937.942415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573938.006365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573938.006375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573938.006379:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573938.006387:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2154 00000100:00080000:2.0:1788573938.006396:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573938.006402:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573938.006432:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495567616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573938.006454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573938.006457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573938.006458:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573938.006461:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2154 00000100:00080000:2.0:1788573938.006477:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573938.006481:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573938.006489:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573938.006497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495567616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573938.006502:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495567744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573938.006513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573938.006515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573938.006516:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573938.006520:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2154 00000100:00080000:2.0:1788573938.006522:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573938.006524:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573938.006527:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573938.006535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495567872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573938.006542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573938.006545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573938.006546:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573938.006549:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2154 00000100:00080000:2.0:1788573938.006553:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573938.006556:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573938.006559:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573938.006568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495568000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573938.006574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573938.006613:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff93a249408e00 x1875453495567616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573938.006628:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495567616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573938.006636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573938.006639:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573938.006642:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495567616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573938.006659:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573938.006677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573938.006685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff93a24940b800 x1875453495567744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573938.006693:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495567744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573938.006698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573938.006701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573938.006703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573938.006709:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567872:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573938.006740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573938.006752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff93a249409c00 x1875453495567872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573938.006765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495567872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573938.006771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573938.006773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573938.006776:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495567872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573938.006780:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495568000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573938.006805:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573938.006813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff93a24940a680 x1875453495568000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573938.006820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495568000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573938.006826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573938.006828:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573938.006830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495568000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573938.966342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573938.966433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573939.990309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573939.990451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573941.014331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573941.014399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573942.038304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573942.038385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573943.062394:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573943.062450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573943.126351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573943.126363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573943.126470:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573943.126480:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2159 00000100:00080000:2.0:1788573943.126491:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573943.126497:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573943.126532:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495568128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573943.126562:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573943.126565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573943.126566:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573943.126569:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2159 00000100:00080000:2.0:1788573943.126572:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573943.126575:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573943.126582:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573943.126595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495568256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573943.126602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573943.126604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573943.126605:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573943.126608:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2159 00000100:00080000:2.0:1788573943.126611:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573943.126613:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788573943.126627:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495568128:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788573943.126633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573943.126642:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495568384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573943.126667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573943.126670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573943.126671:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573943.126674:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2159 00000100:00080000:2.0:1788573943.126676:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573943.126679:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573943.126682:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573943.126689:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495568512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573943.126743:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573943.126816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff93a249409180 x1875453495568128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573943.126840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495568128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573943.126853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573943.126858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573943.126863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495568128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573943.126872:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573943.126898:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573943.126912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff93a249408a80 x1875453495568256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573943.126924:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495568256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573943.126934:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573943.126938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573943.126941:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573943.126973:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568384:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573943.126996:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573943.127007:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff93a24940ad80 x1875453495568384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573943.127035:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495568384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573943.127044:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573943.127047:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573943.127051:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568384:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573943.127057:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495568512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573943.127078:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573943.127089:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff93a249408000 x1875453495568512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573943.127100:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495568512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573943.127108:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573943.127112:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573943.127115:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495568512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573944.086360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573944.086455:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573945.110355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573945.110493:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573946.134345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573946.134432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573947.158376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573947.158474:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573948.182373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573948.182449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573948.246380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573948.246395:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573948.246400:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573948.246411:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2164 00000100:00080000:1.0:1788573948.246457:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573948.246465:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573948.246512:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495568640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573948.246544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573948.246548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573948.246549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573948.246552:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2164 00000100:00080000:1.0:1788573948.246571:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573948.246575:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573948.246586:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573948.246598:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495568768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573948.246604:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495568640:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573948.246608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573948.246612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573948.246614:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573948.246617:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2164 00000100:00080000:1.0:1788573948.246620:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573948.246622:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573948.246626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573948.246635:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495568896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573948.246643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573948.246646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573948.246647:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573948.246650:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2164 00000100:00080000:1.0:1788573948.246654:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573948.246656:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573948.246659:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573948.246665:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495569024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573948.246682:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573948.246734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff93a258da0a80 x1875453495568640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573948.246759:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495568640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573948.246768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573948.246772:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573948.246775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495568640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573948.246809:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573948.246829:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573948.246838:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff93a258da0700 x1875453495568768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573948.246847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495568768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573948.246854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573948.246856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573948.246859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573948.246862:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568896:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573948.246875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573948.246882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff93a258da1c00 x1875453495568896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573948.246890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495568896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573948.246905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573948.246908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573948.246910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495568896:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573948.246914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495569024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573948.246928:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573948.246934:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff93a258da1880 x1875453495569024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573948.246942:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495569024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573948.246947:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573948.246949:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573948.246951:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495569024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573949.206347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573949.206472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573950.230353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573950.230551:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573951.254329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573951.254395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573952.278323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573952.278395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573953.302305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573953.302380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573953.366342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573953.366352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573953.366356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573953.366364:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2169 00000100:00080000:1.0:1788573953.366373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573953.366378:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573953.366411:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495569152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573953.366435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573953.366437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573953.366439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573953.366442:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2169 00000100:00080000:1.0:1788573953.366444:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573953.366446:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573953.366454:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573953.366463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495569280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573953.366471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573953.366473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573953.366475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573953.366479:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495569152:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788573953.366495:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2169 00000100:00080000:1.0:1788573953.366499:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573953.366502:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573953.366524:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573953.366535:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495569408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573953.366543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573953.366545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573953.366547:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573953.366550:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2169 00000100:00080000:1.0:1788573953.366553:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573953.366556:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573953.366559:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573953.366561:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788573953.366570:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495569536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573953.366600:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff93a258da3100 x1875453495569152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.366614:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495569152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.366627:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.366631:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573953.366635:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495569152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573953.366659:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573953.366697:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573953.366708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff93a258da2a00 x1875453495569280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.366716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495569280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.366722:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.366725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573953.366727:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573953.366731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569408:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573953.366743:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573953.366749:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff93a258da3480 x1875453495569408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.366756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495569408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.366761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.366763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573953.366765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573953.366776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495569536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573953.366789:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573953.366796:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff93a258da0380 x1875453495569536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573953.366803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495569536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573953.366808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573953.366809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573953.366811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495569536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573954.326315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573954.326389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573955.350388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573955.350469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573956.374362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573956.374424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573957.398319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573957.398441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573958.422306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573958.422356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573958.486305:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573958.486313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573958.486317:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573958.486325:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2174 00000100:00080000:0.0:1788573958.486333:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573958.486339:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573958.486373:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495569664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573958.486400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573958.486404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573958.486405:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573958.486410:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2174 00000100:00080000:0.0:1788573958.486414:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573958.486420:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573958.486430:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573958.486442:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495569664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788573958.486448:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495569792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573958.486484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573958.486488:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573958.486496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573958.486502:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2174 00000100:00080000:0.0:1788573958.486507:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573958.486513:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788573958.486515:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573958.486524:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573958.486572:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495569920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573958.486574:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff93a2460d2a00 x1875453495569664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573958.486595:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573958.486601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573958.486603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573958.486609:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2174 00000100:00100000:3.0:1788573958.486611:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495569664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573958.486615:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573958.486623:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573958.486631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788573958.486634:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573958.486642:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788573958.486647:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495570048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573958.486648:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495569664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573958.486682:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573958.486704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573958.486714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff93a2460d1180 x1875453495569792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573958.486723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495569792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573958.486729:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573958.486731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573958.486734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573958.486738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569920:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573958.486750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573958.486756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff93a2460d2300 x1875453495569920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573958.486763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495569920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573958.486769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573958.486771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573958.486773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495569920:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573958.486776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495570048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573958.486786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573958.486792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff93a2460d1f80 x1875453495570048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573958.486806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495570048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573958.486811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573958.486814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573958.486816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495570048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573959.446292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573959.446346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573960.470325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573960.470414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573961.494322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573961.494376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573962.518331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573962.518381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573963.542312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573963.542383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573963.606360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573963.606384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573963.606388:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573963.606396:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2180 00000100:00080000:2.0:1788573963.606405:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573963.606411:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573963.606439:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495570176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573963.606459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573963.606462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573963.606463:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573963.606467:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2180 00000100:00080000:2.0:1788573963.606470:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573963.606473:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573963.606481:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573963.606487:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495570176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788573963.606492:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495570304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573963.606501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573963.606503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573963.606504:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573963.606507:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2180 00000100:00080000:2.0:1788573963.606509:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573963.606511:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573963.606515:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573963.606525:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495570432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573963.606538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573963.606540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573963.606541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573963.606545:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2180 00000800:00000100:3.0:1788573963.606550:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788573963.606552:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573963.606557:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573963.606562:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573963.606571:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495570560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573963.606594:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff93a24940aa00 x1875453495570176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573963.606624:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495570176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573963.606632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573963.606635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573963.606638:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495570176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573963.606656:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573963.606675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573963.606684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff93a249408700 x1875453495570304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573963.606699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495570304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573963.606705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573963.606707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573963.606709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573963.606713:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570432:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573963.606727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573963.606733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff93a249409880 x1875453495570432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573963.606741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495570432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573963.606746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573963.606748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573963.606749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570432:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573963.606753:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495570560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573963.606763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573963.606768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff93a249409500 x1875453495570560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573963.606775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495570560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573963.606789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573963.606791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573963.606794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495570560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573964.566372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573964.566446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573965.590332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573965.590462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573966.614372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573966.614443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573967.638320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573967.638464:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573968.662516:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573968.663348:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573968.726353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573968.726363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573968.726367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573968.726375:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2185 00000100:00080000:0.0:1788573968.726400:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573968.726406:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573968.726436:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495570688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573968.726459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573968.726462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573968.726463:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573968.726466:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2185 00000100:00080000:0.0:1788573968.726468:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573968.726470:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573968.726478:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573968.726487:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495570816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573968.726495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573968.726500:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573968.726501:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495570688:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573968.726502:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573968.726508:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2185 00000100:00080000:0.0:1788573968.726513:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573968.726515:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573968.726519:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573968.726528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495570944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573968.726535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573968.726544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573968.726546:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573968.726549:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2185 00000100:00080000:0.0:1788573968.726553:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573968.726555:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573968.726559:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573968.726567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495571072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573968.726574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573968.726613:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff93a2460d1f80 x1875453495570688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573968.726628:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495570688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573968.726636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573968.726639:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573968.726642:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495570688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573968.726675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573968.726693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573968.726708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff93a2460d2300 x1875453495570816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573968.726718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495570816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573968.726729:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573968.726732:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573968.726734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573968.726739:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570944:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573968.726755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573968.726762:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff93a2460d1180 x1875453495570944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573968.726769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495570944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573968.726775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573968.726777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573968.726779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495570944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573968.726782:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495571072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573968.726792:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573968.726798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff93a2460d2a00 x1875453495571072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573968.726804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495571072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573968.726809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573968.726811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573968.726818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495571072:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788573969.686612:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573969.686651:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788573970.710352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573970.710411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573971.734319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573971.734388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573972.758618:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573972.759087:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573973.782602:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573973.782856:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573973.846342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573973.846354:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573973.846359:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573973.846370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2190 00000100:00080000:0.0:1788573973.846381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573973.846389:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573973.846422:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495571200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573973.846446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573973.846449:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573973.846450:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573973.846453:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2190 00000100:00080000:0.0:1788573973.846456:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573973.846459:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573973.846466:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573973.846475:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495571328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573973.846498:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495571200:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573973.846500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573973.846507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573973.846510:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573973.846514:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2190 00000100:00080000:0.0:1788573973.846517:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573973.846519:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573973.846523:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573973.846532:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495571456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573973.846539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573973.846542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573973.846544:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573973.846546:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2190 00000100:00080000:0.0:1788573973.846548:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573973.846550:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573973.846553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573973.846561:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495571584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573973.846570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573973.846619:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff93a2460d3480 x1875453495571200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573973.846640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495571200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573973.846655:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573973.846660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573973.846664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495571200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573973.846692:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573973.846735:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573973.846750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff93a2460d0380 x1875453495571328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573973.846762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495571328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573973.846770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573973.846774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573973.846785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573973.846793:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571456:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573973.846817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573973.846829:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff93a2460d2680 x1875453495571456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573973.846841:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495571456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573973.846849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573973.846852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573973.846856:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571456:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573973.846861:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495571584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573973.846879:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573973.846889:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff93a2460d1500 x1875453495571584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573973.846901:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495571584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573973.846909:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573973.846912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573973.846916:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495571584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573974.806344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573974.806449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573975.830283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573975.830381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573976.854472:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573976.854546:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573977.878364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573977.879135:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573978.902310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573978.902372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573978.966327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573978.966336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573978.966339:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573978.966348:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2195 00000100:00080000:0.0:1788573978.966372:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573978.966377:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573978.966405:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495571712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573978.966428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573978.966431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573978.966432:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573978.966435:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2195 00000100:00080000:0.0:1788573978.966437:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573978.966439:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573978.966446:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573978.966455:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495571840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573978.966462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573978.966464:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573978.966466:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573978.966469:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2195 00000100:00080000:0.0:1788573978.966471:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573978.966475:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495571712:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573978.966476:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573978.966484:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573978.966494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495571968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573978.966501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573978.966509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573978.966511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573978.966516:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2195 00000100:00080000:0.0:1788573978.966523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573978.966527:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573978.966533:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573978.966549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495572096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573978.966549:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573978.966594:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff93a2460d0000 x1875453495571712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573978.966609:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495571712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573978.966616:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573978.966619:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573978.966622:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495571712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573978.966628:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573978.966644:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573978.966652:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff93a2460d3b80 x1875453495571840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573978.966660:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495571840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573978.966665:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573978.966667:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573978.966669:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573978.966687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571968:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573978.966710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573978.966718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff93a2460d0700 x1875453495571968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573978.966725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495571968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573978.966731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573978.966733:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573978.966735:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495571968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573978.966738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495572096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573978.966782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573978.966789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff93a2460d0a80 x1875453495572096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573978.966797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495572096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573978.966802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573978.966804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573978.966807:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495572096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573979.926348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573979.926404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573980.950297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573980.950372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573981.974722:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573981.974774:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573982.998358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573982.998444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573984.022374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573984.022460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573984.086353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573984.086361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573984.086364:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573984.086373:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2200 00000100:00080000:2.0:1788573984.086381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573984.086387:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573984.086420:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495572224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573984.086444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573984.086446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573984.086448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573984.086451:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2200 00000100:00080000:2.0:1788573984.086453:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573984.086455:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573984.086462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573984.086476:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495572352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573984.086491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573984.086496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573984.086498:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573984.086511:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2200 00000100:00100000:3.0:1788573984.086518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495572224:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573984.086519:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573984.086524:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573984.086557:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573984.086578:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495572480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573984.086593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573984.086596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573984.086598:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573984.086602:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2200 00000100:00080000:2.0:1788573984.086618:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573984.086625:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573984.086631:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573984.086643:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573984.086647:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495572608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573984.086710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff93a24940b480 x1875453495572224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573984.086728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495572224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573984.086738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573984.086741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573984.086745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495572224:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573984.086751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573984.086795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573984.086806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff93a24940bb80 x1875453495572352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573984.086815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495572352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573984.086822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573984.086824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573984.086826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573984.086851:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572480:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573984.086864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573984.086871:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff93a24940b100 x1875453495572480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573984.086890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495572480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573984.086896:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573984.086898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573984.086901:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572480:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573984.086904:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495572608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573984.086918:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573984.086929:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff93a249409500 x1875453495572608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573984.086941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495572608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573984.086950:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573984.086954:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573984.086958:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495572608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573985.046324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573985.046929:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573986.070384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573986.070464:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573987.094320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573987.094385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573988.118330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573988.118388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573989.142570:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573989.144434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573989.206342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573989.206353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573989.206358:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573989.206370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2205 00000100:00080000:0.0:1788573989.206384:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573989.206392:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788573989.206443:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495572736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573989.206471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573989.206475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573989.206476:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573989.206479:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2205 00000100:00080000:0.0:1788573989.206482:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573989.206485:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573989.206496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573989.206506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495572864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573989.206533:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573989.206536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573989.206537:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573989.206542:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2205 00000100:00080000:0.0:1788573989.206545:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573989.206548:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573989.206552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573989.206562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495572992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573989.206574:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495572736:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788573989.206580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573989.206588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573989.206591:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573989.206597:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2205 00000100:00080000:0.0:1788573989.206603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573989.206609:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788573989.206616:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573989.206628:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495573120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573989.206656:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573989.206702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff93a2460d0a80 x1875453495572736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573989.206736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495572736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573989.206748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573989.206752:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573989.206755:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495572736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573989.206762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573989.206815:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573989.206828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff93a2460d0700 x1875453495572864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573989.206841:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495572864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573989.206850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573989.206854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573989.206858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573989.206884:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572992:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573989.206902:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573989.206911:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff93a2460d3b80 x1875453495572992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573989.206923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495572992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573989.206947:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573989.206952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573989.206956:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495572992:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573989.206963:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495573120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573989.206981:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573989.206995:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff93a2460d0000 x1875453495573120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573989.207007:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495573120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573989.207015:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573989.207018:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573989.207022:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495573120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573990.166325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573990.166413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573991.190301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573991.190357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573992.214353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573992.214416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573993.238358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573993.238447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573994.262443:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573994.262911:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573994.326391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573994.326407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573994.326413:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573994.326428:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2210 00000100:00080000:1.0:1788573994.326443:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573994.326453:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788573994.326506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495573248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573994.326542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573994.326547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573994.326549:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573994.326554:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2210 00000100:00080000:1.0:1788573994.326578:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573994.326585:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573994.326599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573994.326606:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495573248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788573994.326621:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495573376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573994.326637:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573994.326642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573994.326644:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573994.326651:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2210 00000100:00080000:1.0:1788573994.326656:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573994.326661:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573994.326669:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573994.326685:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495573504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573994.326698:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573994.326703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573994.326707:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573994.326714:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2210 00000100:00080000:1.0:1788573994.326720:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573994.326726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495573376:192.168.202.102@tcp:38:kworker.0 00000100:00080000:1.0:1788573994.326729:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788573994.326739:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573994.326762:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495573632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573994.326772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495573504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573994.326819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495573632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573995.286809:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573995.287069:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573996.310407:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788573996.311127:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573997.334616:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788573997.335180:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788573997.398449:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788573997.398488:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573997.398502:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573997.398551:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573997.398561:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788573997.398570:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788573997.398607:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff93a258da0380 x1875453495573632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573997.398633:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495573632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573997.398646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573997.398651:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573997.398656:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495573632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573997.398666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff93a258da0000 x1875453495573376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573997.398680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495573376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573997.398687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573997.398690:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573997.398692:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495573376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573997.398697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff93a258da2300 x1875453495573504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573997.398706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495573504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573997.398711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573997.398720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573997.398722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495573504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573997.398730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff93a258da1180 x1875453495573248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573997.398744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495573248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573997.398753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573997.398756:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573997.398759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495573248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573998.358391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573998.358413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573999.382358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788573999.382441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573999.446369:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573999.446385:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573999.446413:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573999.446433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2215 00000100:00080000:2.0:1788573999.446445:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573999.446453:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788573999.446493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495573760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573999.446526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573999.446530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573999.446531:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573999.446535:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2215 00000100:00080000:2.0:1788573999.446538:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573999.446541:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573999.446554:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573999.446568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495573888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573999.446579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573999.446583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573999.446585:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573999.446590:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2215 00000100:00080000:2.0:1788573999.446593:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573999.446600:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495573760:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788573999.446601:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573999.446612:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573999.446624:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495574016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573999.446651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573999.446656:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573999.446666:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573999.446671:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2215 00000100:00080000:2.0:1788573999.446691:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573999.446696:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788573999.446703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573999.446718:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788573999.446722:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495574144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573999.446799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff93a249409880 x1875453495573760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573999.446861:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495573760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573999.446875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573999.446881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573999.446886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495573760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788573999.446896:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495573888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573999.446927:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573999.446956:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff93a249408700 x1875453495573888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573999.446985:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495573888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573999.446996:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573999.447000:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573999.447004:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495573888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788573999.447048:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574016:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788573999.447077:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573999.447091:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff93a24940aa00 x1875453495574016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573999.447286:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495574016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573999.447296:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573999.447300:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573999.447304:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574016:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788573999.447310:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495574144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788573999.447335:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788573999.447349:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff93a249408000 x1875453495574144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573999.447363:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495574144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573999.447379:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573999.447383:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788573999.447387:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495574144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574000.406333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574000.406371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574001.430294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574001.430347:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574002.454311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574002.454354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574003.478334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574003.478360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574004.502344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574004.502367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574004.566308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.566316:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574004.566320:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.566329:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2221 00000100:00080000:2.0:1788574004.566338:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.566344:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574004.566374:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495574272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574004.566394:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.566397:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574004.566398:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.566402:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2221 00000100:00080000:2.0:1788574004.566404:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.566406:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574004.566425:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574004.566439:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495574400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574004.566446:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495574272:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574004.566457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.566462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574004.566464:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.566469:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2221 00000100:00080000:2.0:1788574004.566474:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.566479:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574004.566485:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574004.566499:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495574528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574004.566507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.566510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574004.566512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574004.566515:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574004.566519:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2221 00000100:00080000:2.0:1788574004.566523:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.566527:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574004.566532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574004.566540:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495574656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574004.566562:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff93a24940ad80 x1875453495574272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574004.566580:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495574272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574004.566590:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574004.566604:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574004.566612:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495574272:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574004.566634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574004.566655:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574004.566666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff93a249408a80 x1875453495574400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574004.566679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495574400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574004.566695:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574004.566699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574004.566702:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574004.566708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574528:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574004.566730:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574004.566741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff93a249409180 x1875453495574528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574004.566754:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495574528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574004.566762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574004.566766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574004.566769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574004.566776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495574656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574004.566808:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574004.566819:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff93a24940a680 x1875453495574656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574004.566845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495574656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574004.566852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574004.566855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574004.566862:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495574656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574005.526471:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574005.526684:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574006.550370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574006.550402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574007.574316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574007.574335:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574008.598297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574008.598389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574009.622329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574009.622364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574009.686385:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574009.686394:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574009.686397:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574009.686405:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2226 00000100:00080000:0.0:1788574009.686414:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574009.686420:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574009.686453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495574784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574009.686479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574009.686482:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574009.686484:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574009.686487:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2226 00000100:00080000:0.0:1788574009.686489:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574009.686491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574009.686499:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574009.686507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495574912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574009.686516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574009.686520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574009.686522:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574009.686530:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2226 00000100:00100000:3.0:1788574009.686531:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495574784:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574009.686553:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574009.686559:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574009.686566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574009.686581:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495575040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574009.686591:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574009.686599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574009.686606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574009.686608:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574009.686612:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2226 00000100:00080000:0.0:1788574009.686616:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574009.686620:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574009.686625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574009.686635:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495575168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574009.686654:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff93a2460d1500 x1875453495574784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574009.686674:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495574784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574009.686683:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574009.686686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574009.686689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495574784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574009.686696:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574009.686714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574009.686723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff93a2460d2680 x1875453495574912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574009.686731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495574912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574009.686737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574009.686754:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574009.686756:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495574912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574009.686776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575040:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574009.686796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574009.686809:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff93a2460d0380 x1875453495575040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574009.686821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495575040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574009.686830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574009.686834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574009.686837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575040:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574009.686843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495575168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574009.686875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574009.686883:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff93a2460d3480 x1875453495575168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574009.686891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495575168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574009.686896:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574009.686898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574009.686901:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495575168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574010.646304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574010.646379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574011.670319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574011.670405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574012.694297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574012.694368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574013.718309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574013.718412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574014.742293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574014.742397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574014.806416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574014.806428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574014.806433:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574014.806445:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2231 00000100:00080000:0.0:1788574014.806458:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574014.806467:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574014.806532:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495575296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574014.806557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574014.806560:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574014.806562:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574014.806565:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2231 00000100:00080000:0.0:1788574014.806569:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574014.806573:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574014.806583:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574014.806595:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495575424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574014.806600:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495575296:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574014.806605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574014.806612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574014.806614:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574014.806617:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2231 00000100:00080000:0.0:1788574014.806621:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574014.806624:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574014.806630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574014.806643:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495575552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574014.806655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574014.806659:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574014.806662:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574014.806673:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2231 00000100:00080000:0.0:1788574014.806680:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574014.806684:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574014.806689:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574014.806695:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574014.806701:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495575680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574014.806755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff93a2460d2a00 x1875453495575296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574014.806778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495575296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574014.806790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574014.806795:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574014.806800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495575296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574014.806829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574014.806855:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574014.806899:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff93a2460d1180 x1875453495575424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574014.806914:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495575424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574014.806924:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574014.806927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574014.806931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574014.806938:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575552:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574014.806969:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574014.806981:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff93a2460d2300 x1875453495575552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574014.806992:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495575552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574014.807000:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574014.807004:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574014.807008:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575552:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574014.807014:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495575680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574014.807032:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574014.807043:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff93a2460d1f80 x1875453495575680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574014.807054:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495575680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574014.807063:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574014.807066:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574014.807069:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495575680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574015.766350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574015.766424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574016.790372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574016.790445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574017.814341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574017.814407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574018.838321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574018.838375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574019.862304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574019.862390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574019.926371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574019.926399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574019.926403:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574019.926414:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2236 00000100:00080000:3.0:1788574019.926425:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574019.926433:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574019.926468:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495575808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574019.926504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574019.926508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574019.926511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574019.926515:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2236 00000100:00080000:3.0:1788574019.926519:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574019.926522:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574019.926532:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574019.926544:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495575936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574019.926553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574019.926556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574019.926558:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574019.926563:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2236 00000100:00080000:3.0:1788574019.926567:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574019.926571:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574019.926577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574019.926587:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495576064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574019.926591:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495575808:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574019.926611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574019.926615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574019.926617:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574019.926623:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2236 00000100:00080000:3.0:1788574019.926627:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574019.926632:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574019.926637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574019.926649:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495576192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574019.926661:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574019.926703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff93a25064e680 x1875453495575808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574019.926722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495575808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574019.926733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574019.926737:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574019.926742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495575808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574019.926749:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574019.926772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574019.926784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff93a25064fb80 x1875453495575936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574019.926812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495575936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574019.926821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574019.926825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574019.926828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495575936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574019.926854:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576064:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574019.926874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574019.926885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff93a25064f480 x1875453495576064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574019.926914:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495576064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574019.926922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574019.926925:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574019.926929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574019.926935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495576192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574019.926954:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574019.926965:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff93a25064c000 x1875453495576192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574019.926977:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495576192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574019.926997:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574019.927001:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574019.927005:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495576192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574020.886321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574020.886381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574021.910317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574021.910388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574022.934390:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574022.934439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574023.958304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574023.958353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574024.982328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574024.982383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574025.046330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574025.046339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574025.046359:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574025.046367:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2241 00000100:00080000:2.0:1788574025.046376:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574025.046382:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574025.046412:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495576320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574025.046438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574025.046441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574025.046442:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574025.046445:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2241 00000100:00080000:2.0:1788574025.046447:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574025.046449:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574025.046457:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574025.046466:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495576448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574025.046473:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574025.046476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574025.046485:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574025.046488:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2241 00000100:00080000:2.0:1788574025.046491:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574025.046496:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495576320:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574025.046496:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574025.046504:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574025.046512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495576576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574025.046531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574025.046534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574025.046535:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574025.046539:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2241 00000100:00080000:2.0:1788574025.046542:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574025.046545:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574025.046548:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574025.046558:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495576704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574025.046565:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574025.046608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff93a249409c00 x1875453495576320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574025.046626:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495576320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574025.046637:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574025.046641:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574025.046645:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495576320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574025.046652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574025.046674:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574025.046685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff93a24940b800 x1875453495576448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574025.046698:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495576448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574025.046706:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574025.046710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574025.046713:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574025.046744:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576576:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574025.046764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574025.046774:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff93a249408e00 x1875453495576576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574025.046786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495576576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574025.046794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574025.046797:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574025.046801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576576:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574025.046807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495576704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574025.046824:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574025.046833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff93a24940a680 x1875453495576704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574025.046845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495576704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574025.046853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574025.046856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574025.046859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495576704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574026.006350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574026.006370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574027.030364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574027.030421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574028.054331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574028.054418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574029.078315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574029.078374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574030.102316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574030.102365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574030.166312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.166320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574030.166323:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.166331:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2246 00000100:00080000:3.0:1788574030.166339:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574030.166346:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574030.166390:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495576832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574030.166412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.166415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574030.166416:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.166419:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2246 00000100:00080000:3.0:1788574030.166422:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574030.166425:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574030.166432:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574030.166442:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495576960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574030.166449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.166452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574030.166453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.166456:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2246 00000100:00080000:3.0:1788574030.166459:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574030.166464:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495576832:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574030.166465:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574030.166474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574030.166489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495577088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574030.166502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.166506:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574030.166508:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.166519:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2246 00000100:00080000:3.0:1788574030.166523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574030.166528:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788574030.166528:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574030.166536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574030.166547:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495577216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574030.166577:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff93a25064d180 x1875453495576832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.166600:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495576832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.166608:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.166612:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574030.166615:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495576832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574030.166621:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574030.166639:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574030.166647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff93a25064f100 x1875453495576960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.166655:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495576960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.166660:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.166662:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574030.166664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495576960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574030.166684:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577088:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574030.166696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574030.166702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff93a25064ce00 x1875453495577088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.166709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495577088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.166724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.166726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574030.166728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577088:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574030.166732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495577216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574030.166745:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574030.166751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff93a25064e300 x1875453495577216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.166758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495577216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.166763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.166765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574030.166774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495577216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574031.126306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574031.126385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574032.150300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574032.150341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574033.174347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574033.174430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574034.198532:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574034.199201:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574035.222286:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574035.222395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574035.286330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574035.286340:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574035.286344:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574035.286354:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2251 00000100:00080000:2.0:1788574035.286365:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574035.286372:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574035.286409:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495577344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574035.286438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574035.286442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574035.286444:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574035.286448:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2251 00000100:00080000:2.0:1788574035.286452:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574035.286456:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574035.286479:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574035.286494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495577472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574035.286505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574035.286509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574035.286511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574035.286515:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2251 00000100:00080000:2.0:1788574035.286521:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574035.286527:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574035.286530:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495577344:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574035.286533:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574035.286548:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495577600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574035.286560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574035.286564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574035.286566:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574035.286570:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2251 00000100:00080000:2.0:1788574035.286574:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574035.286578:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574035.286583:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574035.286589:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574035.286596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495577728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574035.286635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff93a249409180 x1875453495577344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.286650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495577344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.286659:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.286669:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574035.286673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495577344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574035.286686:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574035.286704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574035.286713:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff93a249408a80 x1875453495577472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.286721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495577472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.286726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.286728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574035.286731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574035.286750:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577600:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574035.286761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574035.286767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff93a24940ad80 x1875453495577600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.286774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495577600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.286789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.286792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574035.286795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574035.286801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495577728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574035.286823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574035.286834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff93a249408000 x1875453495577728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.286854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495577728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.286860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.286862:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574035.286864:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495577728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574036.246328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574036.246384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574037.270298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574037.270375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574038.294366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574038.294421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574039.318497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574039.318846:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574040.342363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574040.342414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574040.406342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574040.406353:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574040.406358:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574040.406369:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2256 00000100:00080000:0.0:1788574040.406380:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574040.406388:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574040.406435:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495577856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574040.406462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574040.406464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574040.406466:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574040.406470:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2256 00000100:00080000:0.0:1788574040.406472:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574040.406474:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574040.406484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574040.406493:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495577984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574040.406500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574040.406503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574040.406505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574040.406508:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2256 00000100:00080000:0.0:1788574040.406510:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574040.406513:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574040.406530:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495577856:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574040.406536:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574040.406547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495578112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574040.406555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574040.406558:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574040.406560:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574040.406562:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2256 00000100:00080000:0.0:1788574040.406565:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574040.406568:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574040.406571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574040.406579:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495578240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574040.406621:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574040.406675:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff93a2460d1f80 x1875453495577856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.406693:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495577856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.406701:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.406705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574040.406708:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495577856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574040.406714:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574040.406733:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574040.406741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff93a2460d2300 x1875453495577984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.406750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495577984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.406766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.406769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574040.406771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495577984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574040.406796:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495578112:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574040.406812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574040.406819:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff93a2460d1180 x1875453495578112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.406827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495578112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.406833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.406835:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574040.406838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495578112:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574040.406841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495578240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574040.406851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574040.406857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff93a2460d2a00 x1875453495578240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.406864:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495578240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.406869:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.406871:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574040.406877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495578240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574041.366341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574041.366392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574042.390353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574042.390440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574043.414330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574043.414423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574044.438374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574044.438432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574045.462345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574045.462430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574045.526335:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574045.526344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574045.526347:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574045.526355:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2261 00000100:00080000:0.0:1788574045.526363:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574045.526370:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574045.526399:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495578368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574045.526442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574045.526445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574045.526447:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574045.526450:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2261 00000100:00080000:0.0:1788574045.526452:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574045.526456:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574045.526463:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574045.526472:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495578496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574045.526482:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574045.526486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574045.526489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574045.526494:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2261 00000100:00080000:0.0:1788574045.526498:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574045.526502:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574045.526508:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495578368:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574045.526510:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574045.526529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495578624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574045.526542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574045.526546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574045.526548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574045.526552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2261 00000100:00080000:0.0:1788574045.526566:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574045.526571:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574045.526580:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574045.526583:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574045.526595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495578752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574045.526625:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff93a2460d3480 x1875453495578368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.526642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495578368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.526657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.526661:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574045.526664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495578368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574045.526669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495578496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574045.526686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574045.526694:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff93a2460d0380 x1875453495578496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.526703:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495578496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.526708:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.526710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574045.526712:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495578496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574045.526730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495578624:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574045.526742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574045.526757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff93a2460d2680 x1875453495578624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.526765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495578624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.526775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.526777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574045.526780:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495578624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574045.526784:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495578752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574045.526797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574045.526803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff93a2460d1500 x1875453495578752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.526811:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495578752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.526815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.526817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574045.526819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495578752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574046.486679:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574046.486742:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574047.510579:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574047.511225:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574048.534367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574048.534452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574049.558382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574049.560118:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574050.582717:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574050.583224:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574050.646399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.646418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574050.646423:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.646435:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2267 00000100:00080000:2.0:1788574050.646447:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.646456:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574050.646503:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495578880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574050.646532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.646536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574050.646537:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.646541:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2267 00000100:00080000:2.0:1788574050.646543:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.646546:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574050.646557:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574050.646568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495579008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574050.646605:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495578880:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574050.646608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.646623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574050.646626:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.646631:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2267 00000100:00080000:2.0:1788574050.646634:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.646638:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574050.646643:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574050.646656:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495579136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574050.646665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.646668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574050.646669:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.646672:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2267 00000100:00080000:2.0:1788574050.646674:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.646677:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574050.646681:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574050.646686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574050.646695:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495579264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574050.646740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff93a24940aa00 x1875453495578880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.646761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495578880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.646783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.646787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574050.646790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495578880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574050.646814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574050.646843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574050.646857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff93a249408700 x1875453495579008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.646886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495579008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.646895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.646899:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574050.646902:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574050.646909:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579136:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574050.646934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574050.646946:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff93a249409880 x1875453495579136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.646957:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495579136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.646965:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.646969:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574050.646972:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579136:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574050.646987:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495579264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574050.647012:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574050.647024:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff93a249409500 x1875453495579264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.647032:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495579264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.647038:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.647040:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574050.647042:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495579264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574051.606656:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574051.606892:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574052.630343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574052.630426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574053.654368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574053.654537:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574054.678320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574054.678391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574055.702428:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574055.702856:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574055.766380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.766388:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574055.766392:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.766402:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2272 00000100:00080000:3.0:1788574055.766413:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.766420:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574055.766452:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495579392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.766492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.766495:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.766496:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.766500:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2272 00000100:00080000:3.0:1788574055.766502:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.766505:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574055.766513:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574055.766523:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495579520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.766530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.766532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.766533:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.766536:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2272 00000100:00080000:3.0:1788574055.766538:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.766540:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574055.766543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574055.766548:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495579392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574055.766553:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495579648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.766563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.766566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.766567:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.766569:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2272 00000100:00080000:3.0:1788574055.766577:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.766579:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574055.766584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574055.766593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495579776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574055.766620:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574055.766673:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff93a25064d500 x1875453495579392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.766694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495579392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.766712:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.766718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574055.766724:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495579392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574055.766733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574055.766758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574055.766772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff93a25064c700 x1875453495579520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.766796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495579520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.766804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.766808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574055.766812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574055.766854:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579648:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574055.766879:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574055.766891:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff93a25064ca80 x1875453495579648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.766904:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495579648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.766912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.766916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574055.766919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495579648:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574055.766926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495579776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574055.766947:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574055.766958:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff93a25064c380 x1875453495579776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.766971:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495579776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.766996:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.767000:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574055.767003:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495579776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574056.726348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574056.726425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574057.750383:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574057.750595:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574058.774601:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574058.775133:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574059.798325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574059.798394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574060.822992:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574060.824190:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788574060.886396:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574060.886408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574060.886413:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574060.886424:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2277 00000100:00080000:0.0:1788574060.886436:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574060.886444:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574060.886489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495579904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574060.886519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574060.886523:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574060.886525:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574060.886530:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2277 00000100:00080000:0.0:1788574060.886534:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574060.886558:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574060.886570:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574060.886581:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495579904:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574060.886585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495580032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574060.886595:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574060.886598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574060.886600:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574060.886604:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2277 00000100:00080000:0.0:1788574060.886608:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574060.886612:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574060.886617:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574060.886626:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495580160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574060.886636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574060.886639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574060.886640:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574060.886644:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2277 00000100:00080000:0.0:1788574060.886647:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574060.886650:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574060.886654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574060.886664:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495580288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574060.886715:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574060.886751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574060.886769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495580288:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574061.846414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574061.846551:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574062.870346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574062.870893:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574063.894691:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574063.895351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:2.0:1788574063.958576:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574063.958630:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574063.958653:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574063.958704:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574063.958714:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574063.958725:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788574063.959052:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574063] req@ffff93a2460d3b80 x1875453495580032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574063.959087:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495580032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574063.959104:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574063.959111:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574063.959116:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574063.959126:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574063] req@ffff93a2460d0700 x1875453495580160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574063.959139:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495580160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574063.959174:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574063.959179:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574063.959183:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574063.959192:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574063] req@ffff93a2460d0a80 x1875453495580288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574063.959205:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495580288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574063.959213:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574063.959217:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574063.959220:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495580288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574063.959266:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574063] req@ffff93a2460d0000 x1875453495579904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574063.959282:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495579904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574063.959292:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574063.959297:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574063.959300:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495579904:192.168.202.102@tcp:250:kworker.0 00010000:00010000:3.0:1788574064.918421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574064.918499:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574065.944846:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574065.944901:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574066.006377:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.006386:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574066.006389:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.006398:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2282 00000100:00080000:3.0:1788574066.006407:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.006413:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574066.006441:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495580416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574066.006462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.006465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574066.006466:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.006469:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2282 00000100:00080000:3.0:1788574066.006489:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.006494:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574066.006504:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574066.006518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495580416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574066.006530:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495580544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574066.006544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.006547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574066.006549:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.006554:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2282 00000100:00080000:3.0:1788574066.006559:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.006563:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574066.006569:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574066.006583:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495580672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574066.006590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788574066.006604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.006609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574066.006611:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.006617:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2282 00000100:00080000:3.0:1788574066.006622:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.006626:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574066.006632:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574066.006644:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495580800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574066.006659:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff93a25064df80 x1875453495580416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.006690:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495580416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.006701:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.006707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574066.006711:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495580416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574066.006740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574066.006782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574066.006796:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff93a25064c380 x1875453495580544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.006809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495580544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.006818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.006821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574066.006825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574066.006831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580672:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574066.006853:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574066.006864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff93a25064ca80 x1875453495580672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.006877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495580672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.006899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.006903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574066.006907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495580672:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574066.006914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495580800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574066.006938:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574066.006950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff93a25064c700 x1875453495580800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.006963:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495580800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.006973:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.006976:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574066.006980:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495580800:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788574066.966815:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574066.966912:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574067.990712:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574067.991171:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574069.014325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574069.014410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574070.038704:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574070.039668:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574071.062296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574071.062352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574071.126308:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.126318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574071.126323:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.126333:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2287 00000100:00080000:3.0:1788574071.126344:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.126351:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574071.126384:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495580928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574071.126407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.126410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574071.126411:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.126414:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2287 00000100:00080000:3.0:1788574071.126417:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.126419:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574071.126428:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574071.126437:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495581056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574071.126445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574071.126450:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495580928:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574071.126450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574071.126453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.126460:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2287 00000100:00080000:3.0:1788574071.126463:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.126465:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574071.126482:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574071.126493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495581184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574071.126501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.126503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574071.126505:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.126507:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2287 00000100:00080000:3.0:1788574071.126510:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.126514:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788574071.126517:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788574071.126523:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574071.126538:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495581312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574071.126569:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff93a25064d500 x1875453495580928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.126593:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495580928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.126604:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.126614:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574071.126617:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495580928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574071.126639:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574071.126675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574071.126690:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff93a25064e300 x1875453495581056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.126716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495581056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.126725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.126727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574071.126730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574071.126735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581184:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574071.126755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574071.126767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff93a25064ce00 x1875453495581184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.126779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495581184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.126787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.126791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574071.126794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574071.126801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495581312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574071.126834:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574071.126847:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff93a25064f100 x1875453495581312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.126858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495581312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.126867:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.126870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574071.126873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495581312:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788574072.087391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574072.087715:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574073.110399:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574073.110507:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574074.134526:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574074.134573:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574075.158658:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574075.160717:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574076.182500:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574076.183036:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:2.0:1788574076.246418:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574076.246430:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574076.246435:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574076.246446:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2292 00000100:00080000:2.0:1788574076.246459:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574076.246467:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574076.246513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495581440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574076.246547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574076.246569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574076.246571:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574076.246575:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2292 00000100:00080000:2.0:1788574076.246578:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574076.246582:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574076.246592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574076.246612:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495581568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574076.246621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574076.246623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574076.246625:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574076.246631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495581440:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574076.246632:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2292 00000100:00080000:2.0:1788574076.246638:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574076.246642:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574076.246646:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574076.246656:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495581696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574076.246663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574076.246709:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574076.246712:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574076.246717:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2292 00000100:00080000:2.0:1788574076.246723:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574076.246728:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574076.246741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574076.246749:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574076.246759:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495581824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574076.246806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff93a24940b100 x1875453495581440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.246828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495581440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.246843:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.246848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574076.246853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495581440:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574076.246882:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574076.246904:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574076.246915:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff93a24940bb80 x1875453495581568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.246932:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495581568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.246938:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.246941:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574076.246943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574076.246948:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581696:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574076.246973:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574076.246981:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff93a24940b480 x1875453495581696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.246989:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495581696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.246995:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.246997:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574076.247000:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495581696:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574076.247004:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495581824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574076.247022:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574076.247029:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff93a249409500 x1875453495581824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.247037:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495581824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.247054:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.247056:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574076.247059:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495581824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574077.206365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574077.206497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574078.230403:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574078.230498:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574079.254373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574079.254416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574080.278322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574080.278370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574081.302316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574081.302416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574081.366360:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574081.366372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574081.366377:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574081.366388:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2297 00000100:00080000:1.0:1788574081.366401:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574081.366408:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574081.366453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495581952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574081.366484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574081.366487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574081.366488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574081.366492:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2297 00000100:00080000:1.0:1788574081.366494:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574081.366497:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574081.366507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574081.366517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495582080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574081.366525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574081.366527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574081.366550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574081.366551:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495581952:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574081.366560:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2297 00000100:00080000:1.0:1788574081.366565:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574081.366572:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574081.366580:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574081.366596:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495582208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574081.366611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574081.366617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788574081.366619:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574081.366622:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574081.366644:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2297 00000100:00080000:1.0:1788574081.366648:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574081.366652:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574081.366658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574081.366673:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495582336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574081.366700:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff93a258da3480 x1875453495581952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574081.366718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495581952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574081.366728:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574081.366731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574081.366736:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495581952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574081.366743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574081.366774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574081.366786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff93a258da2a00 x1875453495582080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574081.366800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495582080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574081.366809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574081.366813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574081.366816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574081.366843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582208:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574081.366861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574081.366870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff93a258da3100 x1875453495582208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574081.366878:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495582208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574081.366883:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574081.366885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574081.366887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582208:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574081.366891:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495582336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574081.366902:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574081.366908:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff93a258da1880 x1875453495582336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574081.366926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495582336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574081.366932:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574081.366934:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574081.366936:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495582336:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788574082.326533:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574082.327014:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574083.350630:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574083.351267:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574084.374326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574084.374416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574085.398447:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574085.398457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574086.422314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574086.422353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574086.486288:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574086.486297:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574086.486301:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574086.486309:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2302 00000100:00080000:1.0:1788574086.486318:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574086.486324:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574086.486372:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495582464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574086.486401:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574086.486405:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574086.486407:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574086.486412:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2302 00000100:00080000:1.0:1788574086.486417:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574086.486422:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574086.486431:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574086.486442:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495582592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574086.486447:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495582464:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574086.486456:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574086.486459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574086.486461:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574086.486464:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2302 00000100:00080000:1.0:1788574086.486468:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574086.486470:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574086.486473:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574086.486482:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495582720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574086.486489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574086.486492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574086.486494:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574086.486502:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2302 00000100:00080000:1.0:1788574086.486506:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574086.486510:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574086.486514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574086.486517:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574086.486527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495582848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574086.486556:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff93a258da1c00 x1875453495582464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574086.486571:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495582464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574086.486579:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574086.486582:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574086.486601:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495582464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574086.486639:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574086.486662:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574086.486673:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff93a258da0700 x1875453495582592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574086.486682:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495582592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574086.486698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574086.486701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574086.486703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574086.486707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582720:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574086.486740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574086.486748:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff93a258da0a80 x1875453495582720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574086.486756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495582720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574086.486761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574086.486763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574086.486765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495582720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574086.486769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495582848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574086.486780:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574086.486785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff93a258da2680 x1875453495582848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574086.486792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495582848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574086.486797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574086.486799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574086.486806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495582848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574087.446317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574087.446384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574088.470372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574088.470446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574089.494347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574089.494366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574090.518329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574090.518394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574091.542385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574091.542486:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574091.606389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574091.606404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574091.606410:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574091.606425:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2308 00000100:00080000:3.0:1788574091.606438:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574091.606447:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574091.606501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495582976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574091.606537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574091.606541:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574091.606543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574091.606560:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2308 00000100:00080000:3.0:1788574091.606565:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574091.606570:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574091.606583:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574091.606586:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495582976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574091.606601:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495583104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574091.606615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574091.606619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574091.606621:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574091.606627:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2308 00000100:00080000:3.0:1788574091.606631:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574091.606656:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788574091.606661:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788574091.606664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574091.606682:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495583232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574091.606694:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574091.606697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574091.606698:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574091.606702:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2308 00000100:00080000:3.0:1788574091.606706:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574091.606710:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574091.606714:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574091.606719:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff93a25064d180 x1875453495582976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574091.606728:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495583360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574091.606748:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495582976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.606762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.606767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574091.606774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495582976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574091.606805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574091.606825:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574091.606835:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff93a25064c000 x1875453495583104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.606844:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495583104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.606851:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.606868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574091.606871:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574091.606876:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583232:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574091.606893:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574091.606901:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff93a25064f480 x1875453495583232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.606909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495583232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.606915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.606917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574091.606919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583232:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574091.606923:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495583360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574091.606934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574091.606939:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff93a25064fb80 x1875453495583360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.606946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495583360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.606952:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.606954:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574091.606956:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495583360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574092.566358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574092.566444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574093.590354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574093.590356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574094.614335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574094.614403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574095.638323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574095.638367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574096.662419:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574096.662443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574096.726360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.726372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574096.726376:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.726387:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2313 00000100:00080000:3.0:1788574096.726400:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.726424:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574096.726475:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495583488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574096.726526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.726532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574096.726575:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.726583:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2313 00000100:00080000:3.0:1788574096.726591:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574096.726593:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495583488:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574096.726599:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574096.726614:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574096.726646:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495583616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574096.726660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.726664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574096.726668:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.726674:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2313 00000100:00080000:3.0:1788574096.726680:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.726687:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788574096.726690:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788574096.726700:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574096.726725:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495583744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574096.726736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff93a25064e680 x1875453495583488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574096.726740:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.726755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574096.726757:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574096.726762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495583488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574096.726770:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2313 00000100:00080000:3.0:1788574096.726786:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574096.726788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.726801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:3.0:1788574096.726801:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574096.726814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495583488:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788574096.726819:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574096.726841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574096.726850:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495583872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574096.726863:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574096.726881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff93a25064ea00 x1875453495583616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.726890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495583616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.726897:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.726900:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574096.726903:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574096.726908:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583744:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574096.726920:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574096.726927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff93a25064dc00 x1875453495583744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.726946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495583744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.726952:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.726954:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574096.726956:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495583744:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574096.726967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495583872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574096.726986:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574096.726994:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff93a25064d880 x1875453495583872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.727002:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495583872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.727007:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.727010:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574096.727012:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495583872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574097.686346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574097.686399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574098.710330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574098.710366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574099.734346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574099.734412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574100.758427:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574100.758444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574101.782314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574101.782408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574101.846328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574101.846335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574101.846339:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574101.846347:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2318 00000100:00080000:0.0:1788574101.846356:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574101.846361:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574101.846394:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495584000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574101.846417:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574101.846421:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574101.846423:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574101.846427:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2318 00000100:00080000:0.0:1788574101.846431:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574101.846437:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574101.846447:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574101.846461:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495584000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574101.846463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495584128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574101.846488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574101.846491:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574101.846492:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574101.846495:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2318 00000100:00080000:0.0:1788574101.846499:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574101.846504:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574101.846508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574101.846532:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495584256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574101.846546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574101.846548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574101.846550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574101.846553:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2318 00000100:00080000:0.0:1788574101.846557:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574101.846560:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574101.846563:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574101.846566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574101.846574:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495584384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574101.846617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff93a2460d0000 x1875453495584000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.846637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495584000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.846657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.846662:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574101.846666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495584000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574101.846688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574101.846712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574101.846724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff93a2460d0a80 x1875453495584128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.846736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495584128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.846744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.846748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574101.846751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574101.846757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584256:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574101.846776:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574101.846786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff93a2460d0700 x1875453495584256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.846799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495584256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.846807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.846811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574101.846814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574101.846827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495584384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574101.846848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574101.846860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff93a2460d3b80 x1875453495584384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.846872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495584384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.846880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.846883:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574101.846886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495584384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574102.806340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574102.806374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574103.830308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574103.830408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574104.854343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574104.854395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574105.878303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574105.878372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574106.902317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574106.902387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574106.966316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574106.966325:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574106.966329:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574106.966337:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2323 00000100:00080000:0.0:1788574106.966372:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574106.966379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574106.966412:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495584512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574106.966435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574106.966451:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574106.966452:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574106.966456:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2323 00000100:00080000:0.0:1788574106.966460:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574106.966464:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574106.966473:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574106.966481:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495584512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574106.966485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495584640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574106.966495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574106.966498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574106.966499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574106.966504:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2323 00000100:00080000:0.0:1788574106.966508:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574106.966512:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574106.966517:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574106.966531:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495584768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574106.966539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574106.966549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574106.966554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574106.966556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574106.966560:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2323 00000100:00080000:0.0:1788574106.966564:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574106.966567:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574106.966577:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574106.966583:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff93a2460d1500 x1875453495584512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574106.966590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495584896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574106.966622:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495584512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.966636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.966640:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574106.966643:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495584512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574106.966662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574106.966681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574106.966691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff93a2460d2680 x1875453495584640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.966707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495584640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.966713:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.966715:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574106.966717:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574106.966721:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584768:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574106.966773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574106.966781:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff93a2460d0380 x1875453495584768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.966789:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495584768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.966795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.966797:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574106.966799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495584768:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574106.966803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495584896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574106.966814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574106.966820:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff93a2460d3480 x1875453495584896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.966827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495584896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.966841:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.966844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574106.966846:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495584896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574107.926291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574107.926360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574108.950367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574108.950435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574109.974373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574109.974439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574110.999152:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574110.999221:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574112.022321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574112.022362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574112.086326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574112.086335:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574112.086339:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574112.086349:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2328 00000100:00080000:0.0:1788574112.086358:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574112.086364:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574112.086399:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495585024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574112.086425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574112.086428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574112.086429:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574112.086432:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2328 00000100:00080000:0.0:1788574112.086434:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574112.086436:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574112.086445:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574112.086453:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495585152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574112.086461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574112.086470:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495585024:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574112.086478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574112.086480:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574112.086484:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2328 00000100:00080000:0.0:1788574112.086487:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574112.086490:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574112.086494:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574112.086502:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495585280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574112.086510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574112.086512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574112.086514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574112.086517:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2328 00000100:00080000:0.0:1788574112.086528:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574112.086530:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574112.086533:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574112.086535:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574112.086543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495585408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574112.086572:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff93a2460d2a00 x1875453495585024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574112.086586:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495585024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574112.086594:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574112.086596:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574112.086599:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495585024:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574112.086617:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574112.086632:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574112.086640:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff93a2460d1180 x1875453495585152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574112.086648:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495585152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574112.086653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574112.086661:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574112.086664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574112.086668:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585280:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574112.086683:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574112.086689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff93a2460d2300 x1875453495585280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574112.086697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495585280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574112.086703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574112.086704:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574112.086706:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574112.086710:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495585408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574112.086721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574112.086727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff93a2460d1f80 x1875453495585408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574112.086734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495585408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574112.086739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574112.086741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574112.086743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495585408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574113.046426:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574113.046877:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574114.070329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574114.070364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574115.094612:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574115.094642:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574116.118524:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574116.118572:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574117.142374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574117.142464:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788574117.206340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.206349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574117.206353:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.206361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2333 00000100:00080000:3.0:1788574117.206371:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.206377:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574117.206408:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495585536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574117.206432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.206434:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574117.206436:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.206441:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2333 00000100:00080000:3.0:1788574117.206444:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.206448:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574117.206458:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574117.206475:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495585664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574117.206479:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495585536:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574117.206497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.206501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574117.206503:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.206521:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2333 00000100:00080000:3.0:1788574117.206525:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.206528:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574117.206533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574117.206537:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574117.206545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495585792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574117.206554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.206557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574117.206558:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.206560:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2333 00000100:00080000:3.0:1788574117.206562:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.206565:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574117.206568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574117.206575:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff93a25064ed80 x1875453495585536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574117.206577:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495585920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574117.206620:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495585536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.206629:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.206633:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574117.206635:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495585536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574117.206656:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574117.206671:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574117.206679:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff93a25064d880 x1875453495585664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.206686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495585664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.206692:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.206694:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574117.206696:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574117.206700:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585792:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574117.206710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574117.206724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff93a25064dc00 x1875453495585792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.206732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495585792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.206737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.206739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574117.206742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495585792:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574117.206745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495585920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574117.206757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574117.206762:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff93a25064ea00 x1875453495585920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.206770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495585920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.206774:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.206776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574117.206778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495585920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574118.166335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574118.166398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574119.190301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574119.190362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574120.214361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574120.214380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574121.238328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574121.238413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574122.262521:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574122.263456:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788574122.326276:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.326284:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574122.326287:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.326295:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2338 00000100:00080000:3.0:1788574122.326304:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.326325:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574122.326356:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495586048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.326380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.326382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574122.326384:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.326386:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2338 00000100:00080000:3.0:1788574122.326390:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.326392:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574122.326400:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574122.326408:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495586176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.326414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.326417:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574122.326418:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.326420:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2338 00000100:00080000:3.0:1788574122.326422:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.326425:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574122.326427:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574122.326434:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495586304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.326441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.326444:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574122.326450:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.326453:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2338 00000100:00100000:1.0:1788574122.326458:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495586048:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574122.326459:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.326466:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574122.326470:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574122.326479:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495586432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574122.326518:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574122.326591:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff93a25064e680 x1875453495586048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.326606:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495586048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.326622:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.326625:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574122.326628:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495586048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574122.326633:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574122.326651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574122.326660:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff93a25064fb80 x1875453495586176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.326668:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495586176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.326673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.326675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574122.326678:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574122.326681:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586304:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574122.326692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574122.326697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff93a25064f480 x1875453495586304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.326704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495586304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.326709:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.326711:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574122.326713:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586304:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574122.326737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495586432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574122.326751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574122.326757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff93a25064c000 x1875453495586432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.326765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495586432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.326770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.326772:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574122.326774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495586432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574123.286340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574123.286898:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574124.310654:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574124.310667:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574125.334577:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574125.334648:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574126.358357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574126.358426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574127.382663:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574127.382739:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:1.0:1788574127.446330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574127.446342:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574127.446347:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574127.446358:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2343 00000100:00080000:1.0:1788574127.446370:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574127.446377:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574127.446421:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495586560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574127.446450:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574127.446453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574127.446454:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574127.446458:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2343 00000100:00080000:1.0:1788574127.446461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574127.446486:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574127.446488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495586560:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574127.446497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574127.446513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495586688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574127.446521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574127.446524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574127.446526:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574127.446529:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2343 00000100:00080000:1.0:1788574127.446533:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574127.446535:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574127.446539:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574127.446547:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495586816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574127.446555:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574127.446558:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574127.446559:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574127.446562:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2343 00000100:00080000:1.0:1788574127.446565:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574127.446567:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574127.446571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574127.446580:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495586944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574127.446616:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574127.446642:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574127.446660:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495586944:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788574128.406558:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574128.406732:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574129.430429:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574129.431771:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574130.454655:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574130.454676:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574130.518486:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574130.518569:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574130.518586:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574130.518648:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574130.518664:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574130.518677:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574130.518703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff93a258da0e00 x1875453495586688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574130.518738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495586688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574130.518771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574130.518777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574130.518782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574130.518790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff93a258da1f80 x1875453495586816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574130.518804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495586816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574130.518811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574130.518815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574130.518818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495586816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574130.518824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff93a258da2680 x1875453495586944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574130.518835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495586944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574130.518842:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574130.518845:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574130.518849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495586944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574130.518856:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff93a258da3800 x1875453495586560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574130.518867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495586560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574130.518882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574130.518886:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574130.518890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495586560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574131.478627:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574131.478675:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574132.502349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574132.502367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574132.566358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.566371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574132.566376:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.566389:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2349 00000100:00080000:3.0:1788574132.566402:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.566410:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574132.566455:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495587072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.566502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.566505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574132.566507:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.566510:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2349 00000100:00080000:3.0:1788574132.566513:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.566517:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574132.566529:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574132.566554:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495587200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.566563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.566582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574132.566608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.566615:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2349 00000100:00080000:3.0:1788574132.566621:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.566626:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574132.566633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574132.566637:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495587072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574132.566650:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495587328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.566663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.566667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574132.566670:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.566674:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2349 00000100:00080000:3.0:1788574132.566685:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.566690:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574132.566697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574132.566711:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495587456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574132.566751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574132.566824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff93a25064d180 x1875453495587072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.566867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495587072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.566881:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.566887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574132.566899:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495587072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574132.566910:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574132.566940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574132.566956:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff93a25064f100 x1875453495587200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.566969:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495587200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.566980:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.566983:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574132.566987:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574132.567031:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587328:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574132.567055:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574132.567095:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff93a25064ce00 x1875453495587328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.567109:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495587328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.567117:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.567120:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574132.567124:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587328:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574132.567131:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495587456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574132.567155:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574132.567169:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff93a25064e300 x1875453495587456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.567182:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495587456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.567191:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.567195:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574132.567198:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495587456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574133.526629:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574133.527399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574134.550304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574134.550369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574135.574595:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574135.574915:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574136.598310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574136.598368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574137.622514:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574137.622594:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574137.686311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.686320:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574137.686323:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.686331:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2354 00000100:00080000:0.0:1788574137.686341:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.686347:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574137.686380:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495587584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.686432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.686436:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574137.686438:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.686443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2354 00000100:00080000:0.0:1788574137.686448:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.686453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574137.686464:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574137.686481:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495587584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574137.686492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495587712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.686506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.686510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574137.686512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.686517:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2354 00000100:00080000:0.0:1788574137.686529:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574137.686535:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574137.686537:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574137.686545:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574137.686561:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495587840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.686570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574137.686573:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff93a2460d1f80 x1875453495587584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574137.686574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574137.686578:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.686586:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2354 00000100:00080000:0.0:1788574137.686596:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574137.686597:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495587584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574137.686602:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574137.686608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788574137.686619:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574137.686623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495587968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574137.686627:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574137.686635:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495587584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574137.686662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574137.686680:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574137.686689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff93a2460d2300 x1875453495587712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574137.686697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495587712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574137.686703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574137.686705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574137.686708:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574137.686711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587840:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574137.686722:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574137.686727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff93a2460d1180 x1875453495587840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574137.686735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495587840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574137.686740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574137.686741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574137.686743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495587840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574137.686752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495587968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574137.686766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574137.686772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff93a2460d2a00 x1875453495587968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574137.686780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495587968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574137.686785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574137.686787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574137.686789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495587968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574138.646302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574138.646373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574139.670809:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574139.670972:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574140.694491:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574140.694843:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574141.718388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574141.719392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574142.742338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574142.742391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574142.806330:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.806338:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574142.806342:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.806350:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2359 00000100:00080000:0.0:1788574142.806360:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.806366:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574142.806400:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495588096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574142.806424:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.806427:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574142.806429:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.806432:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2359 00000100:00080000:0.0:1788574142.806434:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.806437:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574142.806446:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574142.806455:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495588224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574142.806464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574142.806469:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495588096:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574142.806481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574142.806483:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.806488:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2359 00000100:00080000:0.0:1788574142.806491:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.806494:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574142.806499:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574142.806508:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495588352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574142.806516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.806519:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574142.806520:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.806526:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2359 00000800:00000100:1.0:1788574142.806528:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574142.806530:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.806534:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574142.806537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574142.806545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495588480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574142.806568:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff93a2460d3480 x1875453495588096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574142.806583:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495588096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574142.806591:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574142.806594:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574142.806597:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495588096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574142.806615:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574142.806651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574142.806662:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff93a2460d0380 x1875453495588224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574142.806681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495588224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574142.806688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574142.806691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574142.806694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574142.806700:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588352:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574142.806717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574142.806726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff93a2460d2680 x1875453495588352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574142.806736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495588352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574142.806755:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574142.806757:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574142.806760:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588352:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574142.806765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495588480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574142.806781:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574142.806790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff93a2460d1500 x1875453495588480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574142.806799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495588480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574142.806817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574142.806820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574142.806823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495588480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574143.766351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574143.766563:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574144.790601:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574144.790987:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574145.814523:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574145.814564:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574146.838339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574146.838396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574147.862691:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574147.862733:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574147.926327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574147.926336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574147.926340:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574147.926348:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2364 00000100:00080000:0.0:1788574147.926358:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574147.926364:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574147.926411:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495588608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574147.926435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574147.926438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574147.926439:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574147.926442:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2364 00000100:00080000:0.0:1788574147.926445:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574147.926447:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574147.926455:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574147.926462:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495588736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574147.926469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574147.926471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574147.926472:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574147.926475:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2364 00000100:00080000:0.0:1788574147.926476:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574147.926478:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574147.926481:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574147.926487:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495588864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574147.926494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574147.926496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574147.926497:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574147.926505:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2364 00000100:00080000:0.0:1788574147.926507:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574147.926513:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574147.926514:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495588608:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574147.926517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574147.926526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495588992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574147.926570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574147.926604:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff93a2460d3b80 x1875453495588608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.926617:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495588608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.926625:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.926628:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574147.926631:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495588608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574147.926636:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574147.926649:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574147.926656:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff93a2460d0700 x1875453495588736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.926663:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495588736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.926668:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.926670:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574147.926682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574147.926693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588864:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574147.926708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574147.926716:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff93a2460d0a80 x1875453495588864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.926723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495588864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.926729:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.926731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574147.926733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495588864:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574147.926751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495588992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574147.926761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574147.926767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff93a2460d0000 x1875453495588992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.926773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495588992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.926778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.926780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574147.926782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495588992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574148.886320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574148.886367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574149.910310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574149.910367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574150.934302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574150.934345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574151.958330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574151.958876:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574152.982309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574152.982350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574153.046345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574153.046354:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574153.046357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574153.046381:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2369 00000100:00080000:1.0:1788574153.046390:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574153.046396:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574153.046424:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495589120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574153.046448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574153.046451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574153.046452:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574153.046455:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2369 00000100:00080000:1.0:1788574153.046458:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574153.046461:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574153.046468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574153.046478:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495589248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574153.046485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574153.046488:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574153.046489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574153.046492:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2369 00000100:00080000:1.0:1788574153.046494:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574153.046499:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574153.046502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495589120:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574153.046503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574153.046512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495589376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574153.046529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574153.046532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574153.046533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574153.046536:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2369 00000100:00080000:1.0:1788574153.046539:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574153.046542:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574153.046545:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574153.046553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495589504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574153.046571:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574153.046613:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff93a258da0a80 x1875453495589120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574153.046632:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495589120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574153.046642:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574153.046647:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574153.046651:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495589120:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574153.046659:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574153.046681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574153.046694:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff93a258da0700 x1875453495589248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574153.046706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495589248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574153.046714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574153.046725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574153.046729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574153.046752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589376:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574153.046773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574153.046785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff93a258da1c00 x1875453495589376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574153.046798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495589376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574153.046807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574153.046810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574153.046813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589376:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574153.046820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495589504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574153.046839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574153.046848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff93a258da1880 x1875453495589504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574153.046855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495589504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574153.046861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574153.046863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574153.046865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495589504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574154.006338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574154.006384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574155.030369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574155.030442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574156.054322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574156.054402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574157.078367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574157.078415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574158.102368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574158.102440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574158.166337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574158.166350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574158.166355:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574158.166366:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2374 00000100:00080000:1.0:1788574158.166378:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574158.166403:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574158.166438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495589632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574158.166461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574158.166463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574158.166465:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574158.166468:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2374 00000100:00080000:1.0:1788574158.166471:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574158.166474:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574158.166497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574158.166507:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495589760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574158.166518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495589632:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574158.166519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574158.166529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574158.166533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574158.166538:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2374 00000100:00080000:1.0:1788574158.166543:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574158.166548:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574158.166553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574158.166567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495589888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574158.166576:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574158.166579:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574158.166580:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574158.166583:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2374 00000100:00080000:1.0:1788574158.166588:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574158.166591:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574158.166594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574158.166600:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574158.166605:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495590016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574158.166647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff93a258da3100 x1875453495589632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574158.166667:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495589632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574158.166679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574158.166691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574158.166697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495589632:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574158.166720:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574158.166744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574158.166758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff93a258da2a00 x1875453495589760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574158.166771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495589760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574158.166779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574158.166782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574158.166817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574158.166825:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589888:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574158.166866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574158.166880:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff93a258da3480 x1875453495589888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574158.166893:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495589888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574158.166902:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574158.166905:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574158.166909:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495589888:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574158.166916:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495590016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574158.166935:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574158.166947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff93a258da1180 x1875453495590016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574158.166958:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495590016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574158.166967:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574158.166970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574158.166974:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495590016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574159.126357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574159.126411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574160.150682:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574160.150760:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574161.174347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574161.174363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574162.198295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574162.198405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574163.222369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574163.222442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574163.286374:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.286387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574163.286391:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.286435:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2379 00000100:00080000:3.0:1788574163.286445:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.286451:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574163.286480:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495590144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574163.286505:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.286508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574163.286509:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.286512:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2379 00000100:00080000:3.0:1788574163.286514:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.286516:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574163.286524:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574163.286534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495590272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574163.286541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.286544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574163.286545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574163.286551:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495590144:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574163.286551:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2379 00000100:00080000:3.0:1788574163.286554:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.286560:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574163.286565:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574163.286572:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495590400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574163.286588:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.286591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574163.286592:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.286595:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2379 00000100:00080000:3.0:1788574163.286598:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.286601:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574163.286604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574163.286613:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495590528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574163.286620:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574163.286704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff93a25064d500 x1875453495590144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.286719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495590144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.286727:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.286730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574163.286733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495590144:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574163.286752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574163.286773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574163.286782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff93a25064c700 x1875453495590272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.286790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495590272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.286795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.286798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574163.286800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574163.286803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590400:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574163.286815:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574163.286824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff93a25064ca80 x1875453495590400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.286853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495590400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.286861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.286865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574163.286868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590400:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574163.286875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495590528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574163.286898:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574163.286910:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff93a25064c380 x1875453495590528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.286928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495590528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.286936:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.286939:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574163.286942:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495590528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574164.246307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574164.246377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574165.270355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574165.270398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574166.294319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574166.294364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574167.318312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574167.318366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574168.342334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574168.342422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574168.406348:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574168.406356:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574168.406360:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574168.406368:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2384 00000100:00080000:0.0:1788574168.406377:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574168.406383:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574168.406421:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495590656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574168.406453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574168.406456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574168.406458:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574168.406473:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2384 00000100:00080000:0.0:1788574168.406477:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574168.406494:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574168.406503:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574168.406505:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495590656:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574168.406520:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495590784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574168.406535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574168.406547:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574168.406550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574168.406555:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2384 00000100:00080000:0.0:1788574168.406562:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574168.406562:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574168.406569:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574168.406575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574168.406586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495590912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574168.406601:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff93a2460d0000 x1875453495590656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.406602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574168.406609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574168.406613:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574168.406619:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2384 00000100:00080000:0.0:1788574168.406623:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574168.406627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495590656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.406630:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574168.406637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788574168.406653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574168.406655:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495591040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574168.406660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574168.406668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495590656:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574168.406691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574168.406710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574168.406718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff93a2460d0a80 x1875453495590784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.406726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495590784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.406731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.406733:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574168.406736:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574168.406739:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590912:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574168.406750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574168.406755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff93a2460d0700 x1875453495590912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.406762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495590912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.406767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.406769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574168.406771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495590912:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574168.406780:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495591040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574168.406793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574168.406802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff93a2460d3b80 x1875453495591040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.406814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495591040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.406823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.406827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574168.406830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495591040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574169.366350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574169.366403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574170.390306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574170.390463:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574171.414342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574171.414349:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574172.438298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574172.438388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574173.462342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574173.462404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574173.526330:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574173.526339:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574173.526342:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574173.526351:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2389 00000100:00080000:0.0:1788574173.526359:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574173.526365:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574173.526398:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495591168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574173.526423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574173.526426:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574173.526427:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574173.526430:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2389 00000100:00080000:0.0:1788574173.526432:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574173.526434:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574173.526443:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574173.526458:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495591296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574173.526469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574173.526471:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574173.526473:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574173.526475:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2389 00000100:00080000:0.0:1788574173.526478:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574173.526483:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574173.526483:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495591168:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574173.526503:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574173.526513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495591424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574173.526521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574173.526524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574173.526525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574173.526528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2389 00000100:00080000:0.0:1788574173.526533:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574173.526536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574173.526539:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574173.526545:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574173.526548:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495591552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574173.526589:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff93a2460d1500 x1875453495591168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574173.526604:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495591168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574173.526612:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574173.526614:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574173.526617:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495591168:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574173.526622:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574173.526647:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574173.526657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff93a2460d2680 x1875453495591296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574173.526669:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495591296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574173.526678:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574173.526681:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574173.526685:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574173.526711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591424:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574173.526728:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574173.526736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff93a2460d0380 x1875453495591424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574173.526743:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495591424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574173.526749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574173.526751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574173.526753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591424:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574173.526757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495591552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574173.526767:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574173.526772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff93a2460d3480 x1875453495591552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574173.526790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495591552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574173.526795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574173.526797:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574173.526799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495591552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574174.486316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574174.486397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574175.510327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574175.510383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574176.534331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574176.534384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574177.558303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574177.558357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574178.582323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574178.582370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574178.646291:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574178.646300:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574178.646303:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574178.646312:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2395 00000100:00080000:0.0:1788574178.646321:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574178.646326:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574178.646358:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495591680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574178.646389:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574178.646392:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574178.646394:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574178.646397:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2395 00000100:00080000:0.0:1788574178.646413:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574178.646417:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574178.646425:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574178.646438:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495591808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574178.646438:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495591680:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574178.646453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574178.646455:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574178.646456:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574178.646460:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2395 00000100:00080000:0.0:1788574178.646462:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574178.646464:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574178.646467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574178.646476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495591936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574178.646483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574178.646485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574178.646487:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574178.646489:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2395 00000100:00080000:0.0:1788574178.646491:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574178.646493:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574178.646497:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574178.646498:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574178.646507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495592064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574178.646535:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff93a2460d2a00 x1875453495591680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.646550:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495591680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.646559:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.646561:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574178.646564:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495591680:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574178.646581:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574178.646596:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574178.646603:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff93a2460d1180 x1875453495591808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.646611:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495591808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.646635:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.646638:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574178.646640:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574178.646644:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591936:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574178.646660:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574178.646667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff93a2460d2300 x1875453495591936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.646674:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495591936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.646680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.646682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574178.646684:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495591936:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574178.646687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495592064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574178.646697:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574178.646702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff93a2460d1f80 x1875453495592064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.646709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495592064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.646714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.646716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574178.646723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495592064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574179.606306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574179.606391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574180.630324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574180.630420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574181.654353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574181.654434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574182.678338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574182.678417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574183.702330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574183.702378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574183.766320:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574183.766329:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574183.766332:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574183.766341:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2400 00000100:00080000:2.0:1788574183.766351:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574183.766358:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574183.766393:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495592192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574183.766415:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574183.766418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574183.766420:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574183.766423:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2400 00000100:00080000:2.0:1788574183.766425:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574183.766442:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574183.766451:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574183.766460:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495592320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574183.766467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574183.766478:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574183.766480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574183.766483:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2400 00000100:00080000:2.0:1788574183.766486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574183.766489:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574183.766493:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574183.766498:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495592192:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574183.766504:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495592448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574183.766514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574183.766516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574183.766518:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574183.766521:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2400 00000100:00080000:2.0:1788574183.766523:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574183.766526:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574183.766529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574183.766537:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495592576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574183.766563:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574183.766620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff93a249409880 x1875453495592192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.766636:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495592192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.766644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.766647:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574183.766651:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495592192:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574183.766656:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574183.766682:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574183.766691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff93a249408700 x1875453495592320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.766699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495592320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.766705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.766707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574183.766709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574183.766727:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592448:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574183.766739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574183.766755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff93a24940aa00 x1875453495592448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.766763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495592448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.766769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.766771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574183.766773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592448:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574183.766776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495592576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574183.766787:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574183.766793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff93a249408000 x1875453495592576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.766800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495592576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.766804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.766806:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574183.766808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495592576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574184.726337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574184.726428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574185.750345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574185.750412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574186.774339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574186.774396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574187.798301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574187.798367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574188.822413:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574188.822504:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574188.886380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574188.886389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574188.886392:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574188.886401:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2405 00000100:00080000:1.0:1788574188.886410:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574188.886416:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574188.886461:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495592704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574188.886485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574188.886488:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574188.886489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574188.886492:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2405 00000100:00080000:1.0:1788574188.886495:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574188.886498:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574188.886505:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574188.886514:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495592832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574188.886521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574188.886524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574188.886525:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574188.886528:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2405 00000100:00080000:1.0:1788574188.886531:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574188.886533:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495592704:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574188.886536:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574188.886543:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574188.886551:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495592960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574188.886559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574188.886561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574188.886563:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574188.886572:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2405 00000100:00080000:1.0:1788574188.886576:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574188.886578:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574188.886583:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574188.886592:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495593088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574188.886604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574188.886641:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff93a258da2300 x1875453495592704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574188.886655:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495592704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574188.886663:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574188.886666:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574188.886669:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495592704:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574188.886673:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574188.886689:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574188.886701:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff93a258da0000 x1875453495592832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574188.886714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495592832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574188.886723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574188.886726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574188.886730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574188.886754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592960:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574188.886772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574188.886792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff93a258da0380 x1875453495592960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574188.886828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495592960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574188.886837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574188.886841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574188.886843:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495592960:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574188.886848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495593088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574188.886864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574188.886872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff93a258da1180 x1875453495593088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574188.886879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495593088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574188.886884:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574188.886887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574188.886889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495593088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574189.846334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574189.846339:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574190.870291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574190.870362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574191.894317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574191.894398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574192.918547:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574192.918687:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574193.942513:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574193.942750:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574194.006336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574194.006347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574194.006352:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574194.006363:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2410 00000100:00080000:0.0:1788574194.006376:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574194.006383:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574194.006428:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495593216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574194.006454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574194.006457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574194.006459:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574194.006462:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2410 00000100:00080000:0.0:1788574194.006464:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574194.006467:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574194.006476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574194.006485:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495593344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574194.006491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574194.006494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574194.006495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574194.006517:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2410 00000100:00080000:0.0:1788574194.006520:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574194.006526:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574194.006529:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495593216:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574194.006531:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574194.006543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495593472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574194.006551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574194.006553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574194.006554:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574194.006557:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2410 00000100:00080000:0.0:1788574194.006559:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574194.006562:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574194.006565:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574194.006574:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495593600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574194.006643:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574194.006674:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593472:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574194.006690:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495593600:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574194.966636:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574194.966807:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574195.990778:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574195.990832:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574197.014747:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574197.014944:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574197.078561:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574197.078612:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574197.078630:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574197.078679:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574197.078692:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574197.078703:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574197.078750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff93a2460d1180 x1875453495593472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.078783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495593472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.078799:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.078822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574197.078827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593472:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574197.078836:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff93a2460d2a00 x1875453495593600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.078851:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495593600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.078860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.078864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574197.078868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495593600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574197.078876:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff93a2460d1f80 x1875453495593216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.078891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495593216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.078912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.078939:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574197.078944:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495593216:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574197.078954:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff93a2460d2300 x1875453495593344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.078971:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495593344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.078980:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.078984:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574197.078987:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574198.038444:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574198.038527:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574199.062310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574199.062369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574199.126363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574199.126372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574199.126376:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574199.126385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2415 00000100:00080000:1.0:1788574199.126394:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574199.126400:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574199.126429:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495593728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574199.126451:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574199.126454:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574199.126455:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574199.126458:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2415 00000100:00080000:1.0:1788574199.126461:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574199.126463:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574199.126471:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574199.126486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495593856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574199.126498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574199.126508:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495593728:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574199.126526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574199.126529:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574199.126536:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2415 00000100:00080000:1.0:1788574199.126541:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574199.126546:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574199.126553:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574199.126566:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495593984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574199.126575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574199.126581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788574199.126581:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574199.126585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574199.126589:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2415 00000100:00080000:1.0:1788574199.126593:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574199.126597:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574199.126601:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574199.126608:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495594112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574199.126633:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff93a258da3480 x1875453495593728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574199.126653:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495593728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574199.126664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574199.126668:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574199.126673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495593728:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574199.126737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574199.126761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574199.126786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff93a258da2a00 x1875453495593856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574199.126799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495593856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574199.126808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574199.126811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574199.126815:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574199.126821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593984:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574199.126847:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574199.126859:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff93a258da3100 x1875453495593984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574199.126872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495593984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574199.126881:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574199.126884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574199.126887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495593984:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574199.126894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495594112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574199.126916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574199.126930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff93a258da1880 x1875453495594112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574199.126942:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495594112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574199.126957:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574199.126961:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574199.126965:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495594112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574200.086355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574200.086424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574201.110406:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574201.110506:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574202.134311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574202.134355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574203.158299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574203.158333:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574204.182311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574204.182376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574204.246311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574204.246320:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574204.246324:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574204.246333:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2420 00000100:00080000:0.0:1788574204.246343:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574204.246351:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574204.246414:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495594240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574204.246435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574204.246438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574204.246439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574204.246443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2420 00000100:00080000:0.0:1788574204.246446:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574204.246450:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574204.246458:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574204.246470:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495594368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574204.246475:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495594240:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574204.246478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574204.246483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574204.246485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574204.246491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2420 00000100:00080000:0.0:1788574204.246493:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574204.246495:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574204.246498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574204.246506:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495594496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574204.246513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574204.246515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574204.246516:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574204.246524:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2420 00000100:00080000:0.0:1788574204.246528:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574204.246530:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574204.246534:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574204.246542:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495594624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574204.246551:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574204.246595:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff93a2460d3480 x1875453495594240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574204.246614:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495594240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574204.246626:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574204.246631:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574204.246634:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495594240:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574204.246653:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495594368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574204.246677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574204.246691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff93a2460d0380 x1875453495594368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574204.246705:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495594368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574204.246714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574204.246725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574204.246727:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495594368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574204.246744:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495594496:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574204.246763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574204.246772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff93a2460d2680 x1875453495594496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574204.246780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495594496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574204.246786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574204.246788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574204.246790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495594496:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574204.246794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495594624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574204.246804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574204.246810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff93a2460d1500 x1875453495594624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574204.246817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495594624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574204.246829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574204.246832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574204.246836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495594624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574205.206410:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574205.206736:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574206.230307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574206.230370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574207.254335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574207.254355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574208.278322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574208.278382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574209.302501:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574209.303283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788574209.366283:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574209.366292:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574209.366296:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574209.366305:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2425 00000100:00080000:0.0:1788574209.366313:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574209.366319:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574209.366351:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495594752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574209.366377:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574209.366381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574209.366383:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574209.366387:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2425 00000100:00080000:0.0:1788574209.366391:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574209.366395:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574209.366404:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574209.366420:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495594880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574209.366424:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495594752:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574209.366439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574209.366466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574209.366469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574209.366475:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2425 00000100:00080000:0.0:1788574209.366480:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574209.366487:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574209.366488:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574209.366498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574209.366511:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495595008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574209.366520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574209.366523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574209.366525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574209.366528:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff93a2460d3b80 x1875453495594752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574209.366531:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2425 00000100:00080000:0.0:1788574209.366538:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574209.366545:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574209.366550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574209.366554:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495594752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574209.366560:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495595136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574209.366566:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574209.366574:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574209.366579:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495594752:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574209.366598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495594880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574209.366614:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574209.366622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff93a2460d0700 x1875453495594880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574209.366630:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495594880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574209.366636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574209.366638:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574209.366640:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495594880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574209.366643:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595008:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574209.366665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574209.366672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff93a2460d0a80 x1875453495595008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574209.366680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495595008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574209.366685:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574209.366687:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574209.366689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595008:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574209.366692:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495595136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574209.366702:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574209.366708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff93a2460d0000 x1875453495595136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574209.366715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495595136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574209.366720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574209.366722:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574209.366723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495595136:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574210.326362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574210.326363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574211.350329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574211.350395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574212.374341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574212.374413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574213.398365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574213.398495:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574214.422306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574214.422362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574214.486300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.486309:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574214.486313:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.486339:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2430 00000100:00080000:0.0:1788574214.486348:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.486354:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574214.486384:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495595264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574214.486411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.486415:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574214.486417:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.486423:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2430 00000100:00080000:0.0:1788574214.486427:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.486432:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574214.486443:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574214.486453:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495595264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574214.486460:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495595392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574214.486476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.486480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574214.486482:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.486487:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2430 00000100:00080000:0.0:1788574214.486492:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.486496:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574214.486502:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574214.486513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495595520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574214.486523:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574214.486544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.486548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574214.486551:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.486556:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2430 00000100:00080000:0.0:1788574214.486561:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.486567:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574214.486570:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff93a2460d0000 x1875453495595264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574214.486574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574214.486593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495595648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574214.486607:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495595264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574214.486616:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574214.486620:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574214.486623:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495595264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574214.486653:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574214.486675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574214.486686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff93a2460d0a80 x1875453495595392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574214.486695:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495595392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574214.486701:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574214.486703:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574214.486705:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574214.486709:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595520:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574214.486720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574214.486726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff93a2460d0700 x1875453495595520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574214.486733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495595520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574214.486748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574214.486751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574214.486753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595520:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574214.486757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495595648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574214.486769:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574214.486775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff93a2460d3b80 x1875453495595648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574214.486782:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495595648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574214.486787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574214.486789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574214.486791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495595648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574215.446318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574215.446843:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574216.470297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574216.470366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574217.494405:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574217.494484:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574218.518310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574218.518357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574219.542307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574219.543333:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574219.606306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574219.606317:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574219.606322:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574219.606333:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2436 00000100:00080000:0.0:1788574219.606344:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574219.606351:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574219.606385:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495595776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574219.606408:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574219.606411:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574219.606426:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574219.606430:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2436 00000100:00080000:0.0:1788574219.606434:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574219.606439:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574219.606448:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495595776:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574219.606449:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574219.606463:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495595904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574219.606472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574219.606474:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574219.606475:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574219.606479:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2436 00000100:00080000:0.0:1788574219.606481:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574219.606483:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574219.606487:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574219.606494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495596032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574219.606502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574219.606530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788574219.606530:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574219.606535:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574219.606540:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2436 00000100:00080000:0.0:1788574219.606544:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574219.606547:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574219.606550:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574219.606565:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495596160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574219.606587:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff93a2460d1500 x1875453495595776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574219.606602:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495595776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574219.606610:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574219.606612:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574219.606615:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495595776:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574219.606643:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574219.606661:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574219.606671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff93a2460d2680 x1875453495595904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574219.606679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495595904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574219.606684:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574219.606687:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574219.606689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495595904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574219.606693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596032:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574219.606704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574219.606710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff93a2460d0380 x1875453495596032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574219.606718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495596032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574219.606723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574219.606724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574219.606726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596032:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574219.606730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495596160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574219.606739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574219.606745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff93a2460d3480 x1875453495596160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574219.606758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495596160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574219.606764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574219.606766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574219.606768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495596160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574220.566314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574220.566476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574221.590316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574221.591363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574222.614298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574222.614350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574223.638307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574223.638343:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574224.662292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574224.662396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574224.726376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574224.726391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574224.726398:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574224.726413:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2441 00000100:00080000:0.0:1788574224.726428:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574224.726437:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574224.726489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495596288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574224.726531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574224.726536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574224.726538:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574224.726544:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2441 00000100:00080000:0.0:1788574224.726552:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574224.726558:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574224.726571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574224.726582:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495596288:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574224.726592:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495596416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574224.726609:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574224.726633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574224.726637:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574224.726644:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2441 00000800:00000100:3.0:1788574224.726650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574224.726660:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574224.726670:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574224.726679:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574224.726698:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495596544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574224.726711:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff93a2460d2300 x1875453495596288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574224.726720:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574224.726737:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574224.726742:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574224.726751:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2441 00000100:00080000:0.0:1788574224.726756:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574224.726761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495596288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574224.726766:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574224.726779:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788574224.726785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574224.726791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574224.726795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495596288:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574224.726803:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495596672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574224.726831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574224.726866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574224.726881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff93a2460d1f80 x1875453495596416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574224.726897:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495596416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574224.726922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574224.726926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574224.726929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574224.726935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596544:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574224.726957:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574224.726968:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff93a2460d2a00 x1875453495596544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574224.726981:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495596544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574224.726990:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574224.726993:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574224.726997:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596544:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574224.727024:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495596672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574224.727041:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574224.727052:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff93a2460d1180 x1875453495596672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574224.727065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495596672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574224.727073:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574224.727077:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574224.727089:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495596672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574225.686544:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574225.687353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574226.710294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574226.710332:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574227.734300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574227.734389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574228.758346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574228.758387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574229.782391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574229.783364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574229.846401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574229.846413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574229.846417:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574229.846429:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2446 00000100:00080000:1.0:1788574229.846441:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574229.846473:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574229.846514:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495596800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574229.846540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574229.846543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574229.846544:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574229.846548:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2446 00000100:00080000:1.0:1788574229.846551:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574229.846553:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574229.846563:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574229.846573:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495596928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574229.846581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574229.846584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574229.846585:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574229.846590:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2446 00000100:00080000:1.0:1788574229.846594:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574229.846595:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495596800:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574229.846599:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574229.846606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574229.846612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495597056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574229.846619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574229.846623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574229.846629:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574229.846633:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2446 00000100:00080000:1.0:1788574229.846636:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574229.846639:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574229.846644:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574229.846653:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495597184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574229.846676:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574229.846722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff93a258da1c00 x1875453495596800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574229.846752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495596800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574229.846761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574229.846764:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574229.846768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495596800:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574229.846792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574229.846810:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574229.846819:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff93a258da0700 x1875453495596928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574229.846827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495596928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574229.846834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574229.846836:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574229.846838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495596928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574229.846842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597056:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574229.846853:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574229.846869:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff93a258da0a80 x1875453495597056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574229.846877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495597056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574229.846883:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574229.846885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574229.846887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597056:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574229.846890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495597184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574229.846902:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574229.846908:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff93a258da3800 x1875453495597184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574229.846945:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495597184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574229.846951:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574229.846953:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574229.846957:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495597184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574230.806350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574230.806399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574231.830298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574231.830373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574232.854314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574232.854373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574233.878306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574233.878397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574234.902340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574234.902421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574234.966349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574234.966361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574234.966365:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574234.966377:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2451 00000100:00080000:0.0:1788574234.966389:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574234.966397:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574234.966444:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495597312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574234.966471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574234.966474:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574234.966475:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574234.966479:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2451 00000100:00080000:0.0:1788574234.966481:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574234.966484:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574234.966495:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574234.966504:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495597440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574234.966512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574234.966515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574234.966517:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574234.966523:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495597312:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574234.966524:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2451 00000100:00080000:0.0:1788574234.966528:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574234.966534:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574234.966559:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574234.966570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495597568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574234.966579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574234.966582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574234.966583:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574234.966586:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2451 00000100:00080000:0.0:1788574234.966588:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574234.966591:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574234.966594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574234.966601:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495597696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574234.966629:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574234.966685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff93a2460d1180 x1875453495597312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.966724:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495597312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.966745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.966750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574234.966755:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495597312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574234.966785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574234.966811:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574234.966825:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff93a2460d2a00 x1875453495597440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.966839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495597440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.966849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.966853:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574234.966857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574234.966864:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597568:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574234.966884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574234.966895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff93a2460d1f80 x1875453495597568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.966908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495597568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.966917:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.966921:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574234.966925:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574234.966939:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495597696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574234.966960:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574234.966972:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff93a2460d2300 x1875453495597696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.966985:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495597696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.966994:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.966998:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574234.967002:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495597696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574235.926298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574235.926397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574236.950323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574236.950371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574237.974301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574237.975344:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574238.998349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574238.998387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574240.022340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574240.022392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574240.086383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574240.086395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574240.086400:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574240.086412:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2456 00000100:00080000:0.0:1788574240.086426:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574240.086434:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574240.086481:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495597824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574240.086535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574240.086538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574240.086540:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574240.086544:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2456 00000100:00080000:0.0:1788574240.086546:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574240.086550:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574240.086561:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574240.086572:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495597952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574240.086579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574240.086583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574240.086584:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574240.086587:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2456 00000100:00080000:0.0:1788574240.086589:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574240.086591:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574240.086596:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574240.086598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495597824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574240.086606:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495598080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574240.086631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574240.086635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574240.086637:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574240.086641:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2456 00000100:00080000:0.0:1788574240.086650:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574240.086654:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574240.086659:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574240.086670:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495598208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574240.086699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574240.086753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff93a2460d3480 x1875453495597824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574240.086770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495597824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574240.086780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574240.086785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574240.086791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495597824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574240.086801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574240.086826:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574240.086840:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff93a2460d0380 x1875453495597952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574240.086862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495597952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574240.086869:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574240.086871:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574240.086873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495597952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574240.086898:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598080:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574240.086915:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574240.086923:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff93a2460d2680 x1875453495598080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574240.086931:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495598080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574240.086936:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574240.086938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574240.086939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598080:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574240.086943:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495598208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574240.086953:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574240.086959:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff93a2460d1500 x1875453495598208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574240.086966:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495598208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574240.086981:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574240.086983:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574240.086986:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495598208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574241.046332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574241.046426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574242.070378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574242.070450:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574243.094331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574243.094380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574244.118312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574244.118381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574245.142293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574245.142388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574245.206371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574245.206384:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574245.206388:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574245.206400:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2461 00000100:00080000:0.0:1788574245.206413:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574245.206421:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574245.206487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495598336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574245.206519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574245.206522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574245.206524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574245.206528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2461 00000100:00080000:0.0:1788574245.206531:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574245.206550:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574245.206577:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574245.206609:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495598464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574245.206619:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495598336:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574245.206626:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574245.206630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574245.206631:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574245.206636:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2461 00000100:00080000:0.0:1788574245.206640:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574245.206643:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574245.206647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574245.206657:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495598592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574245.206665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574245.206668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574245.206669:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574245.206672:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2461 00000100:00080000:0.0:1788574245.206674:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574245.206676:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574245.206679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574245.206688:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495598720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574245.206703:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574245.206760:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff93a2460d3b80 x1875453495598336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574245.206778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495598336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574245.206787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574245.206790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574245.206793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495598336:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574245.206818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574245.206837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574245.206845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff93a2460d0700 x1875453495598464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574245.206853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495598464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574245.206859:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574245.206862:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574245.206864:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574245.206875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598592:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574245.206891:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574245.206898:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff93a2460d0a80 x1875453495598592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574245.206906:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495598592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574245.206911:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574245.206913:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574245.206915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574245.206918:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495598720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574245.206928:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574245.206934:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff93a2460d0000 x1875453495598720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574245.206942:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495598720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574245.206947:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574245.206948:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574245.206950:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495598720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574246.166300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574246.166354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574247.190308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574247.190419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574248.214350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574248.214416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574249.238315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574249.238522:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574250.262378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574250.262497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574250.326338:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.326352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574250.326358:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.326373:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2466 00000100:00080000:0.0:1788574250.326387:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.326396:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574250.326476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495598848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574250.326510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.326515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574250.326517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.326522:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2466 00000100:00080000:0.0:1788574250.326528:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.326533:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574250.326546:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574250.326561:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495598848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574250.326563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495598976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574250.326579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.326583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574250.326585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.326591:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2466 00000100:00080000:0.0:1788574250.326595:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.326599:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574250.326625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574250.326638:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495599104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574250.326648:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574250.326659:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.326663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574250.326666:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.326672:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2466 00000100:00080000:0.0:1788574250.326687:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.326692:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574250.326699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574250.326708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff93a2460d0000 x1875453495598848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574250.326714:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495599232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574250.326741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495598848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574250.326752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574250.326756:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574250.326761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495598848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574250.326787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574250.326809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574250.326824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff93a2460d0a80 x1875453495598976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574250.326838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495598976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574250.326849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574250.326852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574250.326856:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495598976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574250.326863:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495599104:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574250.326882:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574250.326891:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff93a2460d0700 x1875453495599104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574250.326899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495599104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574250.326914:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574250.326916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574250.326919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495599104:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574250.326924:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495599232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574250.326946:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574250.326958:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff93a2460d3b80 x1875453495599232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574250.326972:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495599232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574250.326981:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574250.326984:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574250.326987:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495599232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574251.286301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574251.286410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574252.310381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574252.310460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574253.334350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574253.334400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574254.358308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574254.358387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574255.382323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574255.382417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574255.446427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574255.446442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574255.446449:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574255.446464:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2471 00000100:00080000:0.0:1788574255.446479:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574255.446490:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574255.446552:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495599360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574255.446585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574255.446590:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574255.446614:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574255.446621:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2471 00000100:00080000:0.0:1788574255.446626:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574255.446631:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574255.446645:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574255.446650:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495599360:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574255.446663:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495599488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574255.446678:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574255.446682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574255.446684:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574255.446689:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2471 00000100:00080000:0.0:1788574255.446693:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574255.446698:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574255.446703:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574255.446716:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495599616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574255.446726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574255.446730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574255.446732:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574255.446739:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2471 00000800:00000100:3.0:1788574255.446740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574255.446746:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574255.446752:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574255.446758:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574255.446777:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495599744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574255.446797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff93a2460d1500 x1875453495599360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.446819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495599360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.446847:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.446852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574255.446857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495599360:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574255.446886:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495599488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574255.446913:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574255.446929:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff93a2460d2680 x1875453495599488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.446943:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495599488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.446952:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.446956:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574255.446960:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495599488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574255.446967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495599616:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574255.447004:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574255.447017:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff93a2460d0380 x1875453495599616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.447030:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495599616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.447040:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.447043:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574255.447047:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495599616:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574255.447061:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495599744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574255.447082:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574255.447095:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff93a2460d3480 x1875453495599744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.447108:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495599744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.447117:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.447121:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574255.447125:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495599744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574256.406388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574256.406461:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574257.430284:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574257.430373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574258.454384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574258.454433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574259.478369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574259.478674:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574260.502855:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574260.503004:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574260.566354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.566363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574260.566367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.566376:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2477 00000100:00080000:2.0:1788574260.566386:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.566394:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574260.566435:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495599872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.566464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.566467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574260.566468:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.566472:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2477 00000100:00080000:2.0:1788574260.566475:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.566480:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574260.566489:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574260.566499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495599872:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574260.566517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495600000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.566526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.566528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574260.566530:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.566533:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2477 00000100:00080000:2.0:1788574260.566544:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.566547:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574260.566551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574260.566559:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495600128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.566566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.566568:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574260.566569:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.566572:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2477 00000100:00080000:2.0:1788574260.566574:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.566576:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574260.566580:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574260.566586:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495600256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574260.566596:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574260.566620:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574260.566639:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495600256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574261.526383:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574261.527219:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574262.550368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574262.551019:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574263.574575:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574263.574851:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:0.0:1788574263.638530:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788574263.638581:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574263.638598:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574263.638640:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574263.638650:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574263.638658:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574263.638698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff93a24940a680 x1875453495600256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574263.638763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495600256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574263.638776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574263.638782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574263.638787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495600256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574263.638795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff93a249408a80 x1875453495600000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574263.638809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495600000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574263.638816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574263.638820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574263.638823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574263.638829:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff93a249409180 x1875453495600128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574263.638840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495600128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574263.638846:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574263.638849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574263.638853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574263.638859:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff93a24940ad80 x1875453495599872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574263.638881:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495599872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574263.638890:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574263.638894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574263.638897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495599872:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574264.598337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574264.598478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574265.622319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574265.622353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574265.686305:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574265.686317:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574265.686320:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574265.686347:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2482 00000100:00080000:2.0:1788574265.686357:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574265.686365:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574265.686400:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495600384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574265.686425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574265.686427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574265.686429:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574265.686432:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2482 00000100:00080000:2.0:1788574265.686435:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574265.686438:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574265.686445:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574265.686459:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495600512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574265.686471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574265.686475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574265.686477:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574265.686483:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495600384:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574265.686485:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2482 00000100:00080000:2.0:1788574265.686492:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574265.686498:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574265.686503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574265.686516:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495600640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574265.686542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574265.686546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574265.686548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574265.686552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2482 00000100:00080000:2.0:1788574265.686557:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574265.686562:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574265.686567:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574265.686577:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574265.686583:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495600768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574265.686624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff93a249408e00 x1875453495600384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574265.686644:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495600384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574265.686655:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574265.686660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574265.686664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495600384:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574265.686671:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574265.686695:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574265.686708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff93a24940b800 x1875453495600512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574265.686721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495600512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574265.686744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574265.686748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574265.686752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574265.686781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600640:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574265.686806:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574265.686818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff93a249409c00 x1875453495600640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574265.686830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495600640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574265.686838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574265.686842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574265.686846:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495600640:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574265.686853:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495600768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574265.686873:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574265.686884:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff93a24940ad80 x1875453495600768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574265.686897:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495600768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574265.686905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574265.686908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574265.686918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495600768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574266.646380:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574266.646439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574267.670310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574267.670381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574268.694319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574268.695318:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574269.718337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574269.718387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574270.742334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574270.742484:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574270.806360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.806369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574270.806372:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.806381:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2487 00000100:00080000:2.0:1788574270.806390:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.806396:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574270.806431:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495600896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574270.806458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.806461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574270.806463:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.806467:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2487 00000100:00080000:2.0:1788574270.806471:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.806477:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574270.806503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574270.806506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495600896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574270.806523:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495601024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574270.806540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.806544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574270.806547:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.806552:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2487 00000100:00080000:2.0:1788574270.806558:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.806564:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574270.806571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574270.806581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495601152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574270.806590:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574270.806592:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574270.806600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574270.806603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.806608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2487 00000100:00080000:2.0:1788574270.806613:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.806617:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574270.806622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574270.806642:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495601280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574270.806651:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff93a249409180 x1875453495600896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.806674:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495600896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.806687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.806692:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574270.806697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495600896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574270.806747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574270.806777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574270.806791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff93a249408a80 x1875453495601024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.806801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495601024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.806808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.806811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574270.806824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574270.806830:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601152:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574270.806865:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574270.806891:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff93a24940a680 x1875453495601152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.806906:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495601152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.806912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.806915:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574270.806917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601152:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574270.806922:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495601280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574270.806940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574270.806948:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff93a249408000 x1875453495601280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.806956:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495601280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.806962:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.806964:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574270.806966:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495601280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574271.766294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574271.766341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574272.790300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574272.790381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574273.814337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574273.814386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574274.838330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574274.838523:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574275.862315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574275.862439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574275.926353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574275.926366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574275.926370:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574275.926380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2492 00000100:00080000:0.0:1788574275.926391:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574275.926415:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574275.926446:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495601408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574275.926469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574275.926473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574275.926474:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574275.926479:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2492 00000100:00080000:0.0:1788574275.926484:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574275.926488:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574275.926497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574275.926513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495601536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574275.926525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574275.926529:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495601408:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574275.926530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574275.926534:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574275.926543:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2492 00000100:00080000:0.0:1788574275.926546:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574275.926548:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574275.926553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574275.926565:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495601664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574275.926576:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574275.926580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574275.926589:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574275.926595:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2492 00000800:00000100:3.0:1788574275.926600:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574275.926603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574275.926611:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574275.926618:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574275.926632:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495601792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574275.926647:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574275] req@ffff93a2460d2300 x1875453495601408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574275.926676:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495601408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574275.926687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574275.926691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574275.926694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495601408:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574275.926717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574275.926740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574275.926752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574275] req@ffff93a2460d1f80 x1875453495601536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574275.926765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495601536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574275.926774:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574275.926777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574275.926781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574275.926787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601664:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574275.926820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574275.926832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574275] req@ffff93a2460d2a00 x1875453495601664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574275.926845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495601664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574275.926853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574275.926856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574275.926860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495601664:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574275.926866:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495601792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574275.926886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574275.926896:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574275] req@ffff93a2460d1180 x1875453495601792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574275.926908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495601792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574275.926916:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574275.926920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574275.926923:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495601792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574276.886312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574276.886337:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574277.910346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574277.910405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574278.934328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574278.934391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574279.958376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574279.958419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574280.982367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574280.982435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574281.046342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574281.046358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574281.046364:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574281.046378:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2497 00000100:00080000:0.0:1788574281.046392:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574281.046402:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574281.046471:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495601920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574281.046507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574281.046512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574281.046514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574281.046521:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2497 00000100:00080000:0.0:1788574281.046526:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574281.046545:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495601920:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574281.046554:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574281.046568:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574281.046589:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495602048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574281.046601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574281.046605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574281.046608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574281.046612:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2497 00000100:00080000:0.0:1788574281.046618:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574281.046624:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574281.046630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574281.046644:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495602176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574281.046657:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574281.046658:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574281.046664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574281.046667:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574281.046673:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2497 00000100:00080000:0.0:1788574281.046679:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574281.046684:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574281.046690:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574281.046705:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495602304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574281.046723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff93a2460d1180 x1875453495601920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574281.046750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495601920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574281.046760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574281.046763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574281.046767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495601920:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574281.046792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574281.046814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574281.046828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff93a2460d2a00 x1875453495602048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574281.046842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495602048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574281.046852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574281.046855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574281.046859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574281.046866:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602176:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574281.046884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574281.046892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff93a2460d1f80 x1875453495602176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574281.046920:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495602176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574281.046926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574281.046929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574281.046931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602176:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574281.046935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495602304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574281.046951:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574281.046958:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff93a2460d2300 x1875453495602304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574281.046966:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495602304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574281.046971:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574281.046990:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574281.046995:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495602304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574282.006339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574282.006391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574283.030369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574283.030388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574284.054343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574284.054404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574285.078318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574285.078370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574286.102335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574286.102411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574286.166414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574286.166428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574286.166457:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574286.166472:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2502 00000100:00080000:2.0:1788574286.166485:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574286.166494:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574286.166542:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495602432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574286.166592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574286.166596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574286.166599:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574286.166604:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2502 00000100:00080000:2.0:1788574286.166609:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574286.166614:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574286.166627:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574286.166644:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495602560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574286.166657:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574286.166664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788574286.166664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495602432:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574286.166668:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574286.166678:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2502 00000100:00080000:2.0:1788574286.166683:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574286.166687:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574286.166694:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574286.166707:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495602688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574286.166733:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574286.166738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574286.166740:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574286.166746:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2502 00000100:00080000:2.0:1788574286.166752:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574286.166759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574286.166760:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574286.166770:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574286.166787:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495602816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574286.166824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff93a24940aa00 x1875453495602432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574286.166869:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495602432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574286.166882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574286.166887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574286.166892:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495602432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574286.166937:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574286.166964:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574286.166979:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff93a249408700 x1875453495602560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574286.166993:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495602560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574286.167002:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574286.167006:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574286.167010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574286.167016:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602688:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574286.167039:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574286.167062:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff93a249409880 x1875453495602688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574286.167075:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495602688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574286.167097:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574286.167101:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574286.167104:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495602688:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574286.167111:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495602816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574286.167134:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574286.167145:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff93a249409500 x1875453495602816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574286.167158:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495602816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574286.167167:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574286.167171:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574286.167175:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495602816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574287.126373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574287.126388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574288.150321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574288.150396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574289.174360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574289.174406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574290.198330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574290.198379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574291.222297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574291.222357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574291.286313:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.286322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574291.286325:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.286334:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2507 00000100:00080000:0.0:1788574291.286343:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.286349:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574291.286380:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495602944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574291.286403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.286407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574291.286409:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.286414:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2507 00000100:00080000:0.0:1788574291.286426:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.286431:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574291.286441:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574291.286455:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495603072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574291.286485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.286487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574291.286489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.286495:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2507 00000100:00080000:0.0:1788574291.286499:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.286503:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574291.286507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574291.286518:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495603200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574291.286529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.286535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574291.286540:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574291.286541:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495602944:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574291.286556:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2507 00000100:00080000:0.0:1788574291.286561:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.286566:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574291.286573:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574291.286589:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495603328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574291.286645:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574291.286707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff93a2460d3480 x1875453495602944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574291.286731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495602944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574291.286756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574291.286761:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574291.286765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495602944:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574291.286775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574291.286802:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574291.286832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff93a2460d0380 x1875453495603072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574291.286847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495603072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574291.286856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574291.286860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574291.286864:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574291.286894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603200:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574291.286916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574291.286927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff93a2460d2680 x1875453495603200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574291.286939:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495603200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574291.286948:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574291.286952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574291.286956:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603200:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574291.286968:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495603328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574291.286987:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574291.286998:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff93a2460d1500 x1875453495603328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574291.287011:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495603328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574291.287019:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574291.287023:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574291.287027:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495603328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574292.246305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574292.246379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574293.270348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574293.270458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574294.294399:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574294.294434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574295.318340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574295.318356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574296.342340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574296.342364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574296.406387:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574296.406403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574296.406410:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574296.406424:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2512 00000100:00080000:3.0:1788574296.406440:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574296.406448:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574296.406494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495603456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574296.406524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574296.406546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574296.406548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574296.406552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2512 00000100:00080000:3.0:1788574296.406556:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574296.406560:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574296.406572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574296.406578:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495603456:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574296.406589:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495603584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574296.406598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574296.406601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574296.406602:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574296.406606:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2512 00000100:00080000:3.0:1788574296.406609:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574296.406612:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574296.406615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574296.406623:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495603712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574296.406632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574296.406635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574296.406636:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574296.406638:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2512 00000100:00080000:3.0:1788574296.406641:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574296.406643:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574296.406652:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574296.406659:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574296.406662:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495603840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574296.406717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff93a25064df80 x1875453495603456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574296.406751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495603456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574296.406761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574296.406765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574296.406768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495603456:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574296.406794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574296.406819:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574296.406834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff93a25064c380 x1875453495603584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574296.406848:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495603584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574296.406864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574296.406867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574296.406870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574296.406875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603712:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574296.406893:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574296.406901:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff93a25064ca80 x1875453495603712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574296.406909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495603712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574296.406915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574296.406917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574296.406919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495603712:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574296.406923:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495603840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574296.406934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574296.406939:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff93a25064c700 x1875453495603840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574296.406946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495603840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574296.406952:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574296.406953:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574296.406960:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495603840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574297.366305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574297.366374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574298.390356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574298.390438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574299.414309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574299.414378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574300.438310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574300.438404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574301.462303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574301.462355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574301.526337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.526348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574301.526353:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.526364:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2517 00000100:00080000:3.0:1788574301.526374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.526382:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574301.526418:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495603968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.526447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.526451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574301.526452:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.526457:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2517 00000100:00080000:3.0:1788574301.526461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.526466:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574301.526477:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574301.526486:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495603968:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574301.526493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495604096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.526525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.526529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574301.526531:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.526536:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2517 00000100:00080000:3.0:1788574301.526541:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.526545:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574301.526552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574301.526558:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574301.526566:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495604224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.526578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.526581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574301.526583:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.526588:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2517 00000100:00080000:3.0:1788574301.526592:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574301.526598:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff93a25064d500 x1875453495603968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574301.526599:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574301.526607:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574301.526621:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495604352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574301.526628:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495603968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.526644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.526648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574301.526652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495603968:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574301.526674:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574301.526694:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574301.526707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff93a25064e300 x1875453495604096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.526721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495604096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.526731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.526735:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574301.526739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574301.526745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604224:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574301.526788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574301.526809:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff93a25064ce00 x1875453495604224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.526818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495604224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.526824:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.526826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574301.526828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604224:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574301.526832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495604352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574301.526847:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574301.526853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff93a25064f100 x1875453495604352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.526861:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495604352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.526866:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.526868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574301.526870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495604352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574302.486300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574302.486352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574303.510306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574303.510363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574304.534290:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574304.534381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574305.558314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574305.558353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574306.582320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574306.582351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574306.646345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.646354:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574306.646357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.646379:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2523 00000100:00080000:3.0:1788574306.646388:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.646394:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574306.646428:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495604480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574306.646451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.646453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574306.646455:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.646458:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2523 00000100:00080000:3.0:1788574306.646460:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.646462:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574306.646470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574306.646479:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495604608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574306.646486:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.646489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574306.646491:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574306.646494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495604480:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574306.646497:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2523 00000100:00080000:3.0:1788574306.646503:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.646506:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574306.646509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574306.646536:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495604736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574306.646544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.646547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574306.646548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.646551:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2523 00000100:00080000:3.0:1788574306.646554:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.646559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788574306.646561:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:3.0:1788574306.646564:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574306.646573:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495604864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574306.646607:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff93a25064d180 x1875453495604480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.646622:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495604480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.646638:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.646643:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574306.646647:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495604480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574306.646674:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574306.646696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574306.646706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff93a25064c000 x1875453495604608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.646715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495604608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.646721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.646723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574306.646725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574306.646730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604736:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574306.646744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574306.646755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff93a25064f480 x1875453495604736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.646768:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495604736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.646777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.646780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574306.646784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495604736:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574306.646798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495604864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574306.646842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574306.646851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff93a25064fb80 x1875453495604864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.646859:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495604864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.646865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.646867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574306.646869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495604864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574307.606314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574307.606360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574308.630297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574308.631327:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574309.654316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574309.654377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574310.678335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574310.678381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574311.702312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574311.702388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574311.766412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574311.766423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574311.766427:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574311.766438:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2528 00000100:00080000:0.0:1788574311.766448:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574311.766455:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574311.766492:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495604992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574311.766521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574311.766525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574311.766527:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574311.766531:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2528 00000100:00080000:0.0:1788574311.766535:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574311.766553:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574311.766564:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574311.766569:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495604992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574311.766580:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495605120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574311.766596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574311.766599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574311.766601:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574311.766607:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2528 00000100:00080000:0.0:1788574311.766612:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574311.766616:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574311.766622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574311.766635:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495605248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574311.766647:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574311.766649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574311.766654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574311.766656:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574311.766662:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2528 00000100:00080000:0.0:1788574311.766674:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574311.766679:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574311.766685:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574311.766699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff93a2460d3b80 x1875453495604992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574311.766700:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495605376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574311.766730:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495604992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.766739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.766743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574311.766746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495604992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574311.766766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574311.766782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574311.766791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff93a2460d0700 x1875453495605120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.766806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495605120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.766812:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.766814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574311.766817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574311.766820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605248:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574311.766834:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574311.766840:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff93a2460d0a80 x1875453495605248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.766848:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495605248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.766879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.766881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574311.766884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605248:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574311.766892:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495605376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574311.766907:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574311.766915:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff93a2460d0000 x1875453495605376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.766922:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495605376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.766937:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.766939:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574311.766941:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495605376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574312.726393:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574312.726429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574313.750295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574313.750389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574314.774302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574314.774346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574315.798310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574315.798393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574316.822367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574316.822400:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788574316.886297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574316.886306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574316.886310:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574316.886319:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2533 00000100:00080000:0.0:1788574316.886328:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574316.886334:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574316.886367:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495605504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574316.886391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574316.886394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574316.886395:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574316.886398:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2533 00000100:00080000:0.0:1788574316.886400:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574316.886402:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574316.886415:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574316.886423:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495605632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574316.886430:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574316.886433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574316.886434:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574316.886438:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2533 00000100:00100000:1.0:1788574316.886455:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495605504:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574316.886456:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574316.886465:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574316.886470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574316.886482:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495605760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574316.886491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574316.886493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574316.886495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574316.886497:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2533 00000100:00080000:0.0:1788574316.886507:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574316.886516:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574316.886518:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574316.886524:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574316.886533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495605888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574316.886556:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff93a2460d0000 x1875453495605504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574316.886570:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495605504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574316.886578:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574316.886581:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574316.886583:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495605504:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574316.886589:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574316.886604:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574316.886612:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff93a2460d0a80 x1875453495605632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574316.886621:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495605632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574316.886630:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574316.886633:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574316.886636:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574316.886669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605760:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574316.886690:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574316.886699:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff93a2460d0700 x1875453495605760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574316.886708:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495605760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574316.886717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574316.886720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574316.886724:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495605760:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574316.886730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495605888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574316.886750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574316.886762:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff93a2460d3b80 x1875453495605888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574316.886774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495605888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574316.886783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574316.886786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574316.886790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495605888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574317.846295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574317.846336:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574318.870337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574318.870402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574319.894332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574319.894377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574320.918335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574320.918383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574321.942316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574321.942358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574322.006332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.006343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574322.006357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.006386:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2538 00000100:00080000:3.0:1788574322.006396:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.006403:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574322.006438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495606016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574322.006462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.006465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574322.006467:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.006470:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2538 00000100:00080000:3.0:1788574322.006473:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.006476:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574322.006484:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574322.006493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495606144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574322.006500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.006503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574322.006504:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.006507:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2538 00000100:00080000:3.0:1788574322.006508:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.006510:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574322.006513:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574322.006521:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495606272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574322.006534:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495606016:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574322.006555:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.006559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574322.006561:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.006567:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2538 00000100:00080000:3.0:1788574322.006572:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.006576:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574322.006582:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574322.006602:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495606400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574322.006603:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574322.006649:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff93a25064e680 x1875453495606016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.006676:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495606016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.006687:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.006691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574322.006695:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495606016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574322.006702:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574322.006726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574322.006739:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff93a25064ea00 x1875453495606144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.006753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495606144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.006762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.006765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574322.006769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574322.006792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606272:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574322.006816:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574322.006828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff93a25064dc00 x1875453495606272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.006840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495606272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.006860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.006864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574322.006867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606272:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574322.006874:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495606400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574322.006894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574322.006905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff93a25064d880 x1875453495606400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.006917:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495606400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.006925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.006928:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574322.006931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495606400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574322.966314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574322.966387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574323.990362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574323.990410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574325.014326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574325.014380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574326.038341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574326.038444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574327.063091:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574327.063217:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788574327.126336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574327.126345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574327.126349:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574327.126357:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2543 00000100:00080000:3.0:1788574327.126366:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574327.126373:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574327.126404:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495606528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574327.126429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574327.126432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574327.126434:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574327.126436:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2543 00000100:00080000:3.0:1788574327.126439:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574327.126455:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574327.126463:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574327.126478:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495606528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574327.126478:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495606656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574327.126494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574327.126497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574327.126500:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574327.126504:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2543 00000100:00080000:3.0:1788574327.126509:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574327.126513:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574327.126519:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574327.126533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495606784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574327.126545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574327.126549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574327.126551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574327.126556:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2543 00000100:00080000:3.0:1788574327.126560:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574327.126564:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574327.126568:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574327.126574:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574327.126580:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495606912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574327.126602:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574327.126622:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495606912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574328.086355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574328.086464:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574329.110302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574329.110353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574330.134457:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574330.134525:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574330.198502:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574330.198546:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574330.198564:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574330.198601:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574330.198611:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574330.198628:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788574330.198651:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff93a25064ea00 x1875453495606912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.198688:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495606912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.198716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.198722:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574330.198727:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495606912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574330.198736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff93a25064d880 x1875453495606656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.198751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495606656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.198759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.198763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574330.198766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574330.198774:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff93a25064dc00 x1875453495606784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.198786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495606784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.198795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.198799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574330.198803:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495606784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574330.198811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff93a25064ed80 x1875453495606528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.198824:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495606528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.198840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.198844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574330.198848:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495606528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574331.158424:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574331.158784:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574332.182386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574332.182428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574332.246363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.246372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574332.246375:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.246396:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2548 00000100:00080000:2.0:1788574332.246405:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.246411:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574332.246440:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495607040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574332.246467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.246470:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574332.246472:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.246477:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2548 00000100:00080000:2.0:1788574332.246480:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.246484:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574332.246494:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574332.246516:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495607168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574332.246521:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495607040:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574332.246541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.246546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574332.246548:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.246553:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2548 00000100:00080000:2.0:1788574332.246557:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.246562:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574332.246568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574332.246582:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495607296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574332.246603:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574332.246613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.246617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574332.246619:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.246626:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2548 00000100:00080000:2.0:1788574332.246631:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.246636:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574332.246642:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574332.246652:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff93a24940b480 x1875453495607040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574332.246660:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495607424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574332.246685:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495607040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.246698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.246703:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574332.246708:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495607040:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574332.246735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574332.246761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574332.246776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff93a24940bb80 x1875453495607168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.246789:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495607168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.246798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.246801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574332.246804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574332.246821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607296:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574332.246857:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574332.246870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff93a24940b100 x1875453495607296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.246883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495607296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.246897:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.246901:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574332.246904:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607296:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574332.246910:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495607424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574332.246932:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574332.246943:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff93a249409500 x1875453495607424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.246954:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495607424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.246962:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.246965:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574332.246968:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495607424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574333.206344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574333.206391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574334.230345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574334.230413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574335.254400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574335.254442:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574336.278353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574336.278402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574337.302337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574337.302405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574337.366293:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.366305:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574337.366310:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.366321:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2553 00000100:00080000:2.0:1788574337.366332:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.366340:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574337.366393:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495607552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.366421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.366425:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574337.366427:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.366444:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2553 00000100:00080000:2.0:1788574337.366450:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574337.366459:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495607552:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574337.366459:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574337.366472:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574337.366490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495607680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.366504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.366508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574337.366510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.366515:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2553 00000100:00080000:2.0:1788574337.366519:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.366523:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574337.366529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574337.366534:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574337.366547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495607808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.366560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.366564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574337.366567:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574337.366571:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff93a249409880 x1875453495607552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574337.366575:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2553 00000100:00080000:2.0:1788574337.366580:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.366589:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574337.366598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574337.366609:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495607552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574337.366615:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495607936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574337.366624:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.366633:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574337.366639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495607552:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574337.366665:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574337.366685:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574337.366695:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff93a249408700 x1875453495607680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.366707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495607680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.366716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.366719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574337.366723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574337.366729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607808:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574337.366751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574337.366771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff93a24940aa00 x1875453495607808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.366780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495607808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.366798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.366801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574337.366803:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495607808:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574337.366808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495607936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574337.366823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574337.366831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff93a249408000 x1875453495607936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.366838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495607936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.366843:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.366846:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574337.366849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495607936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574338.326332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574338.326393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574339.350313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574339.350353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574340.374339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574340.374404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574341.398384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574341.398459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574342.422348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574342.422422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574342.486356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574342.486372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574342.486378:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574342.486393:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2558 00000100:00080000:0.0:1788574342.486407:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574342.486417:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574342.486470:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495608064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574342.486504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574342.486509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574342.486511:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574342.486518:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2558 00000100:00080000:0.0:1788574342.486523:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574342.486538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495608064:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574342.486553:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574342.486566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574342.486585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495608192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574342.486598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574342.486602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574342.486605:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574342.486611:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2558 00000800:00000100:1.0:1788574342.486614:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574342.486619:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574342.486627:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574342.486634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574342.486649:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495608320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574342.486661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574342.486662:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff93a2460d1500 x1875453495608064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574342.486671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574342.486675:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574342.486683:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2558 00000100:00080000:0.0:1788574342.486688:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574342.486693:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495608064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574342.486697:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574342.486705:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788574342.486717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574342.486723:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495608448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574342.486728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574342.486738:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495608064:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574342.486773:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574342.486794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574342.486804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff93a2460d2680 x1875453495608192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.486814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495608192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.486820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.486823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574342.486825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574342.486830:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608320:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574342.486842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574342.486849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff93a2460d0380 x1875453495608320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.486857:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495608320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.486862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.486864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574342.486866:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608320:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574342.486869:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495608448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574342.486879:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574342.486885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff93a2460d3480 x1875453495608448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.486898:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495608448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.486904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.486906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574342.486908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495608448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574343.446344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574343.446358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574344.470356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574344.470383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574345.494334:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574345.494354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574346.518404:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574346.518436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574347.542371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574347.542458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574347.606373:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.606384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574347.606389:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.606400:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2564 00000100:00080000:3.0:1788574347.606412:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.606420:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574347.606463:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495608576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.606490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.606493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574347.606495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.606498:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2564 00000100:00080000:3.0:1788574347.606500:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.606503:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574347.606513:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574347.606523:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495608704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.606548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.606551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788574347.606553:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495608576:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574347.606553:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.606561:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2564 00000100:00080000:3.0:1788574347.606566:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.606568:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574347.606572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574347.606581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495608832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.606588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.606591:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574347.606593:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.606596:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2564 00000100:00080000:3.0:1788574347.606598:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.606600:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574347.606603:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574347.606611:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495608960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574347.606636:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574347.606690:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff93a25064e680 x1875453495608576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.606712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495608576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.606721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.606724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574347.606728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495608576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574347.606753:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574347.606772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574347.606784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff93a25064fb80 x1875453495608704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.606798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495608704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.606823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.606827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574347.606831:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574347.606837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608832:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574347.606856:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574347.606864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff93a25064f480 x1875453495608832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.606872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495608832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.606878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.606880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574347.606882:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495608832:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574347.606886:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495608960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574347.606896:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574347.606902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff93a25064c000 x1875453495608960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.606909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495608960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.606914:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.606916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574347.606923:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495608960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574348.566348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574348.566422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574349.590298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574349.590433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574350.614287:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574350.614443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574351.638386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574351.638456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574352.662333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574352.662361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574352.726401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574352.726413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574352.726418:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574352.726430:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2569 00000100:00080000:2.0:1788574352.726442:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574352.726449:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574352.726492:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495609088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574352.726519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574352.726522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574352.726524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574352.726531:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2569 00000100:00080000:2.0:1788574352.726539:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574352.726543:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574352.726555:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574352.726566:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495609216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574352.726575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574352.726578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574352.726579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574352.726584:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2569 00000100:00100000:1.0:1788574352.726587:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495609088:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574352.726606:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574352.726610:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574352.726615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574352.726627:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495609344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574352.726636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574352.726639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574352.726640:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574352.726642:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2569 00000100:00080000:2.0:1788574352.726644:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574352.726647:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574352.726650:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574352.726658:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495609472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574352.726666:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574352.726715:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff93a24940a680 x1875453495609088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574352.726731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495609088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574352.726740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574352.726744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574352.726748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495609088:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574352.726772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574352.726790:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574352.726799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff93a249408a80 x1875453495609216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574352.726808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495609216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574352.726813:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574352.726815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574352.726817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574352.726821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609344:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574352.726833:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574352.726857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff93a249409180 x1875453495609344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574352.726870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495609344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574352.726880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574352.726884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574352.726887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609344:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574352.726894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495609472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574352.726922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574352.726931:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff93a24940ad80 x1875453495609472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574352.726940:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495609472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574352.726946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574352.726948:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574352.726950:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495609472:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574353.686454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574353.687290:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574354.710393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574354.710395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574355.734401:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574355.734415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574356.758387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574356.758421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574357.782315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574357.782404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574357.846346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574357.846355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574357.846358:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574357.846367:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2574 00000100:00080000:3.0:1788574357.846381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574357.846387:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574357.846419:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495609600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574357.846445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574357.846448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574357.846449:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574357.846452:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2574 00000100:00080000:3.0:1788574357.846454:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574357.846456:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574357.846464:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574357.846474:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495609728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574357.846483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574357.846487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788574357.846488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495609600:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574357.846489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574357.846507:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2574 00000100:00080000:3.0:1788574357.846510:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574357.846513:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574357.846517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574357.846526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495609856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574357.846534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574357.846537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574357.846538:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574357.846541:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2574 00000100:00080000:3.0:1788574357.846543:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574357.846545:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574357.846547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574357.846556:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495609984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574357.846565:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574357.846610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff93a25064d180 x1875453495609600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574357.846639:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495609600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574357.846650:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574357.846654:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574357.846658:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495609600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574357.846682:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574357.846708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574357.846722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff93a25064f100 x1875453495609728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574357.846736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495609728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574357.846745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574357.846749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574357.846752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574357.846759:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609856:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574357.846789:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574357.846803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff93a25064ce00 x1875453495609856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574357.846815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495609856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574357.846837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574357.846841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574357.846845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495609856:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574357.846852:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495609984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574357.846875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574357.846887:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff93a25064e300 x1875453495609984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574357.846899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495609984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574357.846907:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574357.846910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574357.846914:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495609984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574358.806392:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574358.806419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574359.830379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574359.830441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574360.854355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574360.854428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574361.878348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574361.878385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574362.902347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574362.902406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574362.966329:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574362.966350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574362.966354:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574362.966363:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2579 00000100:00080000:3.0:1788574362.966371:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574362.966394:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574362.966426:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495610112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574362.966448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574362.966451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574362.966452:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574362.966455:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2579 00000100:00080000:3.0:1788574362.966458:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574362.966460:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574362.966467:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574362.966478:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495610240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574362.966485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574362.966487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574362.966488:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574362.966491:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2579 00000100:00080000:3.0:1788574362.966493:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574362.966495:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574362.966498:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574362.966506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495610368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574362.966515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574362.966525:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495610112:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574362.966528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574362.966540:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574362.966544:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2579 00000100:00080000:3.0:1788574362.966547:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574362.966550:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574362.966554:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574362.966565:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495610496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574362.966619:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574362.966676:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff93a25064d500 x1875453495610112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574362.966694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495610112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574362.966703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574362.966707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574362.966710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495610112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574362.966729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574362.966749:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574362.966758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff93a25064c700 x1875453495610240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574362.966767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495610240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574362.966773:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574362.966776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574362.966778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574362.966813:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610368:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574362.966828:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574362.966835:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff93a25064ca80 x1875453495610368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574362.966843:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495610368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574362.966849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574362.966851:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574362.966853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610368:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574362.966857:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495610496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574362.966868:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574362.966874:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff93a25064c380 x1875453495610496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574362.966888:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495610496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574362.966894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574362.966896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574362.966898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495610496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574363.926361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574363.926433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574364.950353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574364.950375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574365.974359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574365.974437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574366.998366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574366.998435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574368.022319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574368.022410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574368.086337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574368.086356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574368.086360:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574368.086371:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2584 00000100:00080000:1.0:1788574368.086381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574368.086405:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574368.086444:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495610624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574368.086477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574368.086482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574368.086484:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574368.086490:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2584 00000100:00080000:1.0:1788574368.086495:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574368.086500:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574368.086511:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574368.086512:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495610624:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574368.086532:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495610752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574368.086567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574368.086572:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574368.086575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574368.086580:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574368.086587:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2584 00000100:00080000:1.0:1788574368.086594:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574368.086599:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574368.086606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574368.086611:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff93a258da2680 x1875453495610624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574368.086625:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495610880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574368.086629:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495610624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574368.086661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574368.086663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574368.086667:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:1.0:1788574368.086671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1788574368.086675:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495610624:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574368.086677:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574368.086684:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2584 00000100:00080000:1.0:1788574368.086689:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574368.086694:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574368.086699:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610752:192.168.202.102@tcp:38:kworker.0 00000008:00100000:1.0:1788574368.086700:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574368.086720:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495611008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574368.086724:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574368.086737:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff93a258da1f80 x1875453495610752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574368.086756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495610752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574368.086762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574368.086765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574368.086767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574368.086771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610880:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574368.086786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574368.086792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff93a258da0e00 x1875453495610880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574368.086800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495610880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574368.086805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574368.086807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574368.086809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495610880:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574368.086818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495611008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574368.086828:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574368.086834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff93a258da2680 x1875453495611008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574368.086841:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495611008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574368.086856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574368.086858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574368.086860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495611008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574369.046314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574369.046426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574370.070292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574370.070437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574371.094339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574371.094339:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574372.118345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574372.118380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574373.142363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574373.142371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788574373.206326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.206334:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574373.206344:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.206355:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2589 00000100:00080000:0.0:1788574373.206364:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.206371:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574373.206417:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495611136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.206455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.206459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574373.206461:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.206465:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2589 00000100:00080000:0.0:1788574373.206470:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.206474:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574373.206485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574373.206499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495611136:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574373.206502:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495611264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.206531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.206534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574373.206535:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.206538:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2589 00000100:00080000:0.0:1788574373.206542:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.206545:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574373.206549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574373.206559:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495611392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.206566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574373.206571:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574373.206572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574373.206576:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.206579:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2589 00000100:00080000:0.0:1788574373.206582:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.206584:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574373.206588:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574373.206604:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495611520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574373.206622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff93a2460d2300 x1875453495611136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574373.206637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495611136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574373.206645:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574373.206648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574373.206651:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495611136:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574373.206669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574373.206685:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574373.206693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff93a2460d1f80 x1875453495611264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574373.206701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495611264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574373.206713:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574373.206715:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574373.206717:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574373.206721:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611392:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574373.206736:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574373.206743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff93a2460d2a00 x1875453495611392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574373.206750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495611392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574373.206755:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574373.206757:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574373.206759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611392:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574373.206762:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495611520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574373.206772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574373.206778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff93a2460d1180 x1875453495611520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574373.206785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495611520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574373.206789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574373.206791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574373.206798:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495611520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574374.166319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574374.166346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574375.190319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574375.190360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574376.214341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574376.214415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574377.238338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574377.238346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574378.262333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574378.262359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574378.326338:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574378.326350:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574378.326356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574378.326370:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2594 00000100:00080000:0.0:1788574378.326409:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574378.326418:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574378.326459:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495611648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574378.326488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574378.326491:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574378.326506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574378.326511:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2594 00000100:00080000:0.0:1788574378.326514:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574378.326517:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574378.326528:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574378.326550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495611776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574378.326561:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495611648:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574378.326567:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574378.326573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574378.326575:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574378.326581:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2594 00000100:00080000:0.0:1788574378.326586:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574378.326591:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574378.326597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574378.326609:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495611904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574378.326619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574378.326627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574378.326629:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574378.326633:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2594 00000100:00080000:0.0:1788574378.326637:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574378.326640:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574378.326645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574378.326650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574378.326672:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495612032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574378.326720:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff93a2460d1180 x1875453495611648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574378.326741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495611648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574378.326754:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574378.326758:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574378.326762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495611648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574378.326801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574378.326826:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574378.326839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff93a2460d2a00 x1875453495611776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574378.326851:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495611776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574378.326859:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574378.326863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574378.326874:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574378.326881:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611904:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574378.326902:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574378.326913:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff93a2460d1f80 x1875453495611904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574378.326924:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495611904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574378.326932:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574378.326935:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574378.326938:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495611904:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574378.326944:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495612032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574378.326960:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574378.326970:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff93a2460d2300 x1875453495612032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574378.326981:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495612032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574378.326988:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574378.326991:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574378.326994:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495612032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574379.286309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574379.287352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574380.310356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574380.310433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574381.334343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574381.334345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574382.358342:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574382.360702:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574383.382325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574383.382401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574383.446333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.446345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574383.446349:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.446360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2599 00000100:00080000:0.0:1788574383.446372:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.446379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574383.446425:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495612160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574383.446459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.446463:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574383.446465:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.446470:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2599 00000100:00080000:0.0:1788574383.446474:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.446478:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574383.446488:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574383.446500:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495612288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574383.446527:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495612160:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574383.446529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.446535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574383.446538:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.446543:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2599 00000100:00080000:0.0:1788574383.446562:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.446568:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574383.446574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574383.446588:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495612416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574383.446598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.446601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574383.446604:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574383.446606:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574383.446611:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2599 00000100:00080000:0.0:1788574383.446617:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.446622:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574383.446627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574383.446638:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495612544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574383.446662:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff93a2460d3480 x1875453495612160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574383.446680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495612160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574383.446692:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574383.446706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574383.446710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495612160:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574383.446734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574383.446753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574383.446763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff93a2460d0380 x1875453495612288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574383.446772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495612288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574383.446777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574383.446780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574383.446782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574383.446786:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612416:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574383.446809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574383.446817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff93a2460d2680 x1875453495612416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574383.446825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495612416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574383.446830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574383.446832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574383.446833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612416:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574383.446837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495612544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574383.446847:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574383.446853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff93a2460d1500 x1875453495612544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574383.446860:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495612544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574383.446865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574383.446866:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574383.446868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495612544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574384.406382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574384.406399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574385.430347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574385.430376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574386.454335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574386.454345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574387.478344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574387.478399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574388.502389:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574388.502442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574388.566317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574388.566348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574388.566352:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574388.566360:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2605 00000100:00080000:0.0:1788574388.566369:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574388.566375:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574388.566404:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495612672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574388.566427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574388.566433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574388.566435:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574388.566438:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2605 00000100:00080000:0.0:1788574388.566440:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574388.566444:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574388.566454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574388.566468:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495612800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574388.566472:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495612672:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574388.566489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574388.566493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574388.566495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574388.566501:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2605 00000100:00080000:0.0:1788574388.566504:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574388.566507:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574388.566512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574388.566523:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495612928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574388.566537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574388.566540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574388.566544:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574388.566544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574388.566557:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2605 00000100:00080000:0.0:1788574388.566562:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574388.566565:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574388.566570:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574388.566580:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495613056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574388.566615:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff93a2460d3b80 x1875453495612672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574388.566644:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495612672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574388.566652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574388.566655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574388.566658:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495612672:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574388.566683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574388.566702:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574388.566712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff93a2460d0700 x1875453495612800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574388.566721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495612800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574388.566727:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574388.566729:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574388.566731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574388.566735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612928:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574388.566748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574388.566754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff93a2460d0a80 x1875453495612928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574388.566761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495612928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574388.566777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574388.566780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574388.566782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495612928:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574388.566786:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495613056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574388.566800:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574388.566807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff93a2460d0000 x1875453495613056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574388.566814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495613056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574388.566819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574388.566821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574388.566823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495613056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574389.526321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574389.526394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574390.550372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574390.550455:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574391.574304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574391.574361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574392.598357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574392.598637:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574393.622437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574393.622447:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788574393.686340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574393.686351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574393.686356:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574393.686367:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2610 00000100:00080000:0.0:1788574393.686378:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574393.686386:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574393.686431:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495613184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574393.686459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574393.686462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574393.686463:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574393.686467:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2610 00000100:00080000:0.0:1788574393.686469:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574393.686472:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574393.686497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574393.686512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495613312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574393.686514:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495613184:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574393.686550:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574393.686553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574393.686555:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574393.686560:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2610 00000100:00080000:0.0:1788574393.686564:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574393.686568:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574393.686572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574393.686581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495613440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574393.686592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574393.686595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574393.686596:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574393.686599:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2610 00000100:00080000:0.0:1788574393.686601:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574393.686603:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574393.686606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574393.686613:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495613568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574393.686669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574393.686717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574393.686734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495613568:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574394.646408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574394.646905:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574395.670408:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574395.670816:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574396.694530:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574396.694775:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574396.758468:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574396.758504:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574396.758518:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574396.758552:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574396.758562:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574396.758570:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574396.758612:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff93a2460d0a80 x1875453495613312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574396.758638:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495613312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574396.758650:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574396.758655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574396.758659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574396.758668:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff93a2460d0700 x1875453495613440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574396.758696:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495613440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574396.758707:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574396.758711:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574396.758715:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574396.758723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff93a2460d3b80 x1875453495613568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574396.758737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495613568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574396.758746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574396.758766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574396.758770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495613568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574396.758777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff93a2460d0000 x1875453495613184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574396.758792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495613184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574396.758800:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574396.758803:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574396.758807:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495613184:192.168.202.102@tcp:250:kworker.0 00010000:00010000:3.0:1788574397.718854:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574397.719512:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574398.742312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574398.742393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574398.806367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574398.806382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574398.806408:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574398.806422:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2615 00000100:00080000:0.0:1788574398.806435:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574398.806443:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574398.806483:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495613696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574398.806521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574398.806525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574398.806527:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574398.806533:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2615 00000100:00080000:0.0:1788574398.806537:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574398.806543:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574398.806557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574398.806572:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495613824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574398.806589:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495613696:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574398.806590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574398.806599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574398.806606:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574398.806614:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2615 00000100:00080000:0.0:1788574398.806618:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574398.806623:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574398.806636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574398.806649:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495613952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574398.806676:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574398.806684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788574398.806686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574398.806688:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574398.806698:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2615 00000100:00080000:0.0:1788574398.806719:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574398.806727:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574398.806735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574398.806754:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495614080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574398.806769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff93a2460d1500 x1875453495613696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.806811:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495613696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.806826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.806833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574398.806839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495613696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574398.806882:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574398.806922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574398.806967:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff93a2460d2680 x1875453495613824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.806987:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495613824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.806997:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.807001:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574398.807005:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574398.807012:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613952:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574398.807036:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574398.807049:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff93a2460d0380 x1875453495613952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.807061:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495613952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.807069:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.807073:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574398.807076:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495613952:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574398.807083:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495614080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574398.807113:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574398.807126:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff93a2460d3480 x1875453495614080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.807384:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495614080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.807403:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.807408:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574398.807412:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495614080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574399.766337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574399.766364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574400.790363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574400.790406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574401.814347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574401.814413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574402.838371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574402.838437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574403.862300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574403.862416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574403.926361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.926373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574403.926395:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.926405:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2620 00000100:00080000:1.0:1788574403.926415:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.926423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574403.926454:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495614208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.926476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.926479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574403.926481:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.926484:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2620 00000100:00080000:1.0:1788574403.926486:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.926489:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574403.926496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574403.926505:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495614336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.926512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.926515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574403.926516:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.926518:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2620 00000100:00080000:1.0:1788574403.926520:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.926522:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574403.926526:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574403.926533:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495614208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574403.926534:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495614464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.926558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.926561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574403.926562:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.926566:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2620 00000100:00080000:1.0:1788574403.926569:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.926572:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574403.926576:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574403.926585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495614592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574403.926612:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574403.926665:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff93a258da3800 x1875453495614208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.926691:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495614208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.926702:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.926706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574403.926710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495614208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574403.926738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574403.926779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574403.926794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff93a258da0a80 x1875453495614336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.926818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495614336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.926837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.926841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574403.926845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574403.926852:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614464:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574403.926878:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574403.926891:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff93a258da0700 x1875453495614464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.926905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495614464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.926914:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.926918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574403.926922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614464:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574403.926929:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495614592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574403.926949:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574403.926961:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff93a258da1c00 x1875453495614592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.926975:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495614592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.926983:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.926987:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574403.926998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495614592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574404.886374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574404.886458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574405.910358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574405.910433:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574406.934335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574406.934375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574407.958342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574407.959331:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574408.982314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574408.982376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574409.046345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574409.046353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574409.046356:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574409.046365:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2625 00000100:00080000:1.0:1788574409.046374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574409.046379:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574409.046410:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495614720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574409.046437:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574409.046440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574409.046441:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574409.046444:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2625 00000100:00080000:1.0:1788574409.046446:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574409.046448:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574409.046471:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574409.046481:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495614848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574409.046489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574409.046492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574409.046493:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574409.046496:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2625 00000100:00080000:1.0:1788574409.046499:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574409.046502:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574409.046506:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574409.046508:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495614720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574409.046515:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495614976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574409.046524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574409.046526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574409.046527:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574409.046530:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2625 00000100:00080000:1.0:1788574409.046532:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574409.046534:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574409.046536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574409.046543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495615104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574409.046581:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574409.046624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff93a258da1880 x1875453495614720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574409.046645:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495614720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574409.046666:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574409.046671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574409.046676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495614720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574409.046684:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574409.046712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574409.046727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff93a258da3100 x1875453495614848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574409.046742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495614848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574409.046762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574409.046766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574409.046778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574409.046825:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614976:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574409.046853:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574409.046866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff93a258da2a00 x1875453495614976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574409.046880:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495614976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574409.046890:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574409.046893:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574409.046897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495614976:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574409.046904:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495615104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574409.046924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574409.046935:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff93a258da3480 x1875453495615104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574409.046949:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495615104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574409.046957:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574409.046961:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574409.046964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495615104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574410.006366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574410.006466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574411.030386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574411.030454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574412.054359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574412.054429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574413.078341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574413.078426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574414.102390:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574414.102394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574414.166351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574414.166363:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574414.166367:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574414.166379:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2630 00000100:00080000:2.0:1788574414.166392:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574414.166399:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574414.166448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495615232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574414.166486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574414.166490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574414.166491:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574414.166495:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2630 00000100:00080000:2.0:1788574414.166499:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574414.166529:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574414.166541:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574414.166547:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495615232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574414.166555:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495615360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574414.166564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574414.166567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574414.166569:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574414.166572:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2630 00000100:00080000:2.0:1788574414.166575:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574414.166578:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574414.166584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574414.166596:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495615488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574414.166608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574414.166612:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574414.166615:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574414.166619:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2630 00000100:00080000:2.0:1788574414.166624:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574414.166628:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788574414.166633:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:2.0:1788574414.166637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574414.166651:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495615616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574414.166692:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff93a249409c00 x1875453495615232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574414.166725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495615232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574414.166739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574414.166744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574414.166749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495615232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574414.166777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495615360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574414.166806:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574414.166817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff93a24940b800 x1875453495615360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574414.166826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495615360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574414.166833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574414.166835:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574414.166838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495615360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574414.166842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495615488:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574414.166891:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574414.166905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff93a249408e00 x1875453495615488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574414.166919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495615488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574414.166928:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574414.166932:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574414.166935:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495615488:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574414.166940:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495615616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574414.166957:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574414.166965:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff93a24940ad80 x1875453495615616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574414.166973:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495615616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574414.166979:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574414.166981:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574414.166984:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495615616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574415.126370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574415.126429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574416.150336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574416.150367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574417.174323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574417.174428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574418.198362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574418.198451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574419.222304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574419.222405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574419.286301:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574419.286313:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574419.286318:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574419.286329:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2635 00000100:00080000:2.0:1788574419.286340:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574419.286348:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574419.286397:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495615744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574419.286425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574419.286428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574419.286429:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574419.286433:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2635 00000100:00080000:2.0:1788574419.286435:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574419.286438:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574419.286455:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574419.286466:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495615872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574419.286475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574419.286477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574419.286478:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574419.286482:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2635 00000100:00080000:2.0:1788574419.286504:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574419.286508:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574419.286512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574419.286523:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495615744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574419.286529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495616000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574419.286539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574419.286544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574419.286549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574419.286559:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2635 00000100:00080000:2.0:1788574419.286562:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574419.286564:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574419.286567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574419.286575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495616128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574419.286622:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574419.286677:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff93a249409180 x1875453495615744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574419.286700:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495615744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574419.286711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574419.286716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574419.286720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495615744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574419.286729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495615872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574419.286761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574419.286774:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff93a249408a80 x1875453495615872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574419.286787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495615872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574419.286797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574419.286800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574419.286804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495615872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574419.286829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616000:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574419.286851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574419.286863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff93a24940a680 x1875453495616000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574419.286891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495616000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574419.286899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574419.286903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574419.286907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616000:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574419.286926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495616128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574419.286951:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574419.286964:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff93a249408000 x1875453495616128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574419.286977:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495616128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574419.286986:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574419.286989:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574419.286993:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495616128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574420.246326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574420.246428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574421.270326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574421.270443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574422.294341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574422.294341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574423.318328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574423.318405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574424.342318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574424.342333:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574424.406342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574424.406354:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574424.406358:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574424.406371:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2640 00000100:00080000:0.0:1788574424.406385:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574424.406394:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574424.406448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495616256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574424.406483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574424.406488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574424.406490:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574424.406496:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2640 00000100:00080000:0.0:1788574424.406500:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574424.406505:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574424.406538:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574424.406556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495616384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574424.406570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574424.406570:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495616256:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574424.406578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574424.406581:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574424.406586:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2640 00000100:00080000:0.0:1788574424.406589:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574424.406592:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574424.406597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574424.406607:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495616512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574424.406616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574424.406621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574424.406623:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574424.406628:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2640 00000100:00080000:0.0:1788574424.406632:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574424.406636:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574424.406643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574424.406654:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574424.406659:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495616640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574424.406705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff93a2460d2300 x1875453495616256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574424.406730:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495616256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574424.406739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574424.406743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574424.406746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495616256:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574424.406768:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574424.406785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574424.406794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff93a2460d1f80 x1875453495616384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574424.406802:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495616384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574424.406807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574424.406809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574424.406811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574424.406815:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616512:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574424.406826:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574424.406832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff93a2460d2a00 x1875453495616512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574424.406839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495616512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574424.406853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574424.406856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574424.406858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616512:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574424.406862:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495616640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574424.406875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574424.406882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff93a2460d1180 x1875453495616640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574424.406889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495616640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574424.406894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574424.406896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574424.406898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495616640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574425.366378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574425.366439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574426.390333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574426.390392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574427.414326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574427.414339:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574428.438351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574428.438360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574429.462323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574429.462373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574429.526317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574429.526326:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574429.526345:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574429.526354:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2645 00000100:00080000:0.0:1788574429.526363:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574429.526369:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574429.526399:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495616768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574429.526419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574429.526421:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574429.526422:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574429.526425:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2645 00000100:00080000:0.0:1788574429.526427:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574429.526430:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574429.526438:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574429.526446:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495616896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574429.526453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574429.526456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574429.526457:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574429.526461:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2645 00000100:00080000:0.0:1788574429.526465:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574429.526466:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495616768:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574429.526470:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574429.526478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574429.526490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495617024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574429.526516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574429.526520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574429.526523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574429.526526:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574429.526531:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2645 00000100:00080000:0.0:1788574429.526535:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574429.526539:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574429.526544:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574429.526554:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495617152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574429.526590:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff93a2460d1180 x1875453495616768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574429.526609:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495616768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574429.526625:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574429.526637:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574429.526640:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495616768:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574429.526646:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574429.526680:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574429.526696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff93a2460d2a00 x1875453495616896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574429.526709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495616896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574429.526716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574429.526719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574429.526721:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495616896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574429.526740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617024:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574429.526764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574429.526772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff93a2460d1f80 x1875453495617024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574429.526780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495617024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574429.526785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574429.526788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574429.526790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617024:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574429.526800:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495617152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574429.526813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574429.526820:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff93a2460d2300 x1875453495617152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574429.526828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495617152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574429.526832:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574429.526834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574429.526836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495617152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574430.486376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574430.486381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574431.510319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574431.511306:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574432.534310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574432.534360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574433.558341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574433.558402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574434.582325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574434.582369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574434.646325:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.646333:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574434.646351:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.646360:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2651 00000100:00080000:2.0:1788574434.646369:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.646375:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574434.646402:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495617280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574434.646425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.646427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574434.646434:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.646437:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2651 00000100:00080000:2.0:1788574434.646439:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.646442:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574434.646450:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574434.646459:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495617408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574434.646467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.646470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574434.646473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574434.646474:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495617280:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574434.646477:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2651 00000100:00080000:2.0:1788574434.646484:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.646486:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574434.646490:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574434.646498:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495617536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574434.646515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.646518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574434.646520:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.646523:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2651 00000100:00080000:2.0:1788574434.646529:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.646531:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574434.646535:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574434.646543:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574434.646545:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495617664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574434.646586:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff93a24940aa00 x1875453495617280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574434.646603:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495617280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574434.646614:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574434.646618:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574434.646623:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495617280:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574434.646655:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574434.646679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574434.646692:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff93a249408700 x1875453495617408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574434.646704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495617408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574434.646713:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574434.646717:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574434.646720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574434.646727:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617536:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574434.646745:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574434.646753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff93a249409880 x1875453495617536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574434.646761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495617536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574434.646766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574434.646768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574434.646770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617536:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574434.646774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495617664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574434.646791:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574434.646797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff93a249409500 x1875453495617664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574434.646812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495617664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574434.646817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574434.646819:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574434.646821:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495617664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574435.606330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574435.606387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574436.630354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574436.630429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574437.654328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574437.654381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574438.678326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574438.678375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574439.702395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574439.703332:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574439.766357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.766368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574439.766372:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.766383:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2656 00000100:00080000:2.0:1788574439.766393:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.766401:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574439.766437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495617792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574439.766467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.766472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574439.766474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.766479:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2656 00000100:00080000:2.0:1788574439.766483:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.766487:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574439.766512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574439.766528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495617920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574439.766544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.766548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574439.766550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.766559:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2656 00000100:00100000:3.0:1788574439.766563:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495617792:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574439.766566:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.766579:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574439.766586:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574439.766603:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495618048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574439.766616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.766620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574439.766622:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.766627:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2656 00000100:00080000:2.0:1788574439.766640:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.766646:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574439.766652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574439.766659:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574439.766671:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495618176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574439.766737:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff93a24940b100 x1875453495617792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574439.766770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495617792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574439.766782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574439.766787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574439.766791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495617792:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574439.766821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574439.766848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574439.766862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff93a24940bb80 x1875453495617920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574439.766889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495617920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574439.766899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574439.766903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574439.766907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495617920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574439.766914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618048:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574439.766938:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574439.766950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff93a24940b480 x1875453495618048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574439.766963:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495618048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574439.766971:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574439.766975:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574439.766978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618048:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574439.766985:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495618176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574439.767005:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574439.767017:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff93a249409500 x1875453495618176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574439.767029:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495618176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574439.767051:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574439.767055:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574439.767059:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495618176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574440.726321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574440.726351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574441.750331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574441.750382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574442.774346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574442.774413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574443.798398:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574443.798439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574444.822328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574444.822410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574444.886379:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574444.886395:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574444.886401:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574444.886415:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2661 00000100:00080000:2.0:1788574444.886431:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574444.886440:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574444.886495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495618304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574444.886536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574444.886541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574444.886543:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574444.886548:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2661 00000100:00080000:2.0:1788574444.886552:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574444.886557:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574444.886571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574444.886590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495618432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574444.886602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574444.886606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574444.886608:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574444.886613:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2661 00000100:00080000:2.0:1788574444.886617:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574444.886625:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495618304:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574444.886628:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574444.886644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574444.886688:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495618560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574444.886703:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574444.886704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574444.886711:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574444.886714:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574444.886722:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2661 00000100:00080000:2.0:1788574444.886729:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574444.886735:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574444.886742:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574444.886755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff93a249409880 x1875453495618304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574444.886775:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495618688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574444.886787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495618304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574444.886806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574444.886812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574444.886818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495618304:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574444.886828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574444.886855:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574444.886869:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff93a249408700 x1875453495618432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574444.886883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495618432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574444.886909:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574444.886912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574444.886915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574444.886935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618560:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574444.886953:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574444.886961:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff93a24940aa00 x1875453495618560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574444.886969:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495618560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574444.886974:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574444.886976:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574444.886978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618560:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574444.886983:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495618688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574444.887003:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574444.887015:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff93a249408000 x1875453495618688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574444.887027:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495618688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574444.887035:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574444.887039:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574444.887048:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495618688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574445.846306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574445.846352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574446.870310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574446.870353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574447.894303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574447.894341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574448.918313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574448.918401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574449.942320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574449.942365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574450.006347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574450.006363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574450.006367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574450.006376:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2666 00000100:00080000:0.0:1788574450.006385:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574450.006392:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574450.006428:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495618816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574450.006475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574450.006479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574450.006481:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574450.006486:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2666 00000100:00080000:0.0:1788574450.006491:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574450.006496:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574450.006507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574450.006521:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495618816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574450.006526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495618944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574450.006541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574450.006544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574450.006546:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574450.006550:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2666 00000100:00080000:0.0:1788574450.006554:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574450.006557:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574450.006592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574450.006598:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574450.006604:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495619072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574450.006614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574450.006616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574450.006618:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574450.006620:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2666 00000100:00080000:0.0:1788574450.006624:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574450.006626:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574450.006629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574450.006637:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495619200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574450.006665:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff93a2460d3480 x1875453495618816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574450.006683:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495618816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574450.006691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574450.006696:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574450.006706:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495618816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574450.006727:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574450.006752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574450.006766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff93a2460d0380 x1875453495618944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574450.006781:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495618944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574450.006806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574450.006809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574450.006814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495618944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574450.006820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619072:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574450.006840:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574450.006873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff93a2460d2680 x1875453495619072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574450.006882:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495619072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574450.006888:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574450.006891:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574450.006893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619072:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574450.006897:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495619200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574450.006910:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574450.006933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff93a2460d1500 x1875453495619200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574450.006941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495619200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574450.006945:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574450.006947:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574450.006949:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495619200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574450.966318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574450.966378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574451.990347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574451.990377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574453.014333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574453.014497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574454.038342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574454.038390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574455.062333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574455.062371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574455.126344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574455.126353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574455.126356:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574455.126365:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2671 00000100:00080000:0.0:1788574455.126373:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574455.126380:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574455.126408:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495619328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574455.126448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574455.126450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574455.126452:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574455.126455:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2671 00000100:00080000:0.0:1788574455.126457:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574455.126460:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574455.126467:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574455.126479:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495619456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574455.126487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574455.126489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574455.126490:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574455.126495:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2671 00000100:00100000:3.0:1788574455.126497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495619328:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574455.126498:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574455.126505:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574455.126509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574455.126517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495619584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574455.126523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574455.126526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574455.126528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574455.126530:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2671 00000100:00080000:0.0:1788574455.126538:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574455.126541:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574455.126545:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574455.126553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495619712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574455.126565:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574455.126621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff93a2460d0000 x1875453495619328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574455.126640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495619328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574455.126658:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574455.126663:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574455.126666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495619328:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574455.126689:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574455.126713:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574455.126725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff93a2460d3b80 x1875453495619456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574455.126737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495619456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574455.126745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574455.126748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574455.126751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574455.126756:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619584:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574455.126773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574455.126783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff93a2460d0700 x1875453495619584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574455.126794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495619584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574455.126801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574455.126804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574455.126808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619584:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574455.126820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495619712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574455.126838:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574455.126848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff93a2460d0a80 x1875453495619712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574455.126859:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495619712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574455.126867:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574455.126870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574455.126872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495619712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574456.086361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574456.086362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574457.110316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574457.110423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574458.134320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574458.134370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574459.158364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574459.158610:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574460.182468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574460.182757:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:2.0:1788574460.246426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.246433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574460.246453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.246469:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2676 00000100:00080000:2.0:1788574460.246480:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.246487:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574460.246522:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495619840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.246544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.246547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574460.246548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.246551:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2676 00000100:00080000:2.0:1788574460.246554:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.246557:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574460.246565:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574460.246574:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495619968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.246583:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.246585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574460.246586:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.246591:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2676 00000100:00100000:3.0:1788574460.246592:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495619840:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574460.246593:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.246600:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574460.246604:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574460.246612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495620096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.246630:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.246633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574460.246635:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.246637:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2676 00000100:00080000:2.0:1788574460.246642:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.246645:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574460.246649:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574460.246657:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495620224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574460.246698:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574460.246722:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620096:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574460.246740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495620224:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788574461.206631:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574461.206748:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788574462.230496:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574462.231014:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574463.254459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574463.254736:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:0.0:1788574463.318571:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788574463.318617:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574463.318634:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574463.318677:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574463.318687:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574463.318696:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574463.318742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff93a249408a80 x1875453495619968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.318787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495619968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.318799:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.318804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574463.318808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495619968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574463.318836:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff93a249409180 x1875453495620096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.318851:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495620096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.318860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.318863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574463.318867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620096:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574463.318874:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff93a24940ad80 x1875453495620224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.318885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495620224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.318898:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.318902:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574463.318906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495620224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574463.318911:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff93a24940a680 x1875453495619840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.318922:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495619840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.318929:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.318933:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574463.318936:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495619840:192.168.202.102@tcp:250:kworker.0 00010000:00010000:2.0:1788574464.278488:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574464.278699:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574465.302348:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574465.302398:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:2.0:1788574465.366335:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.366346:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574465.366350:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.366361:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2681 00000100:00080000:2.0:1788574465.366372:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.366389:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574465.366443:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495620352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574465.366466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.366485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574465.366487:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.366490:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2681 00000100:00080000:2.0:1788574465.366495:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.366499:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574465.366507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574465.366532:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495620480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574465.366536:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495620352:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574465.366553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.366556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574465.366558:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.366564:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2681 00000100:00080000:2.0:1788574465.366569:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.366573:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574465.366579:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574465.366594:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495620608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574465.366602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.366606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574465.366608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.366612:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2681 00000800:00000100:3.0:1788574465.366614:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574465.366618:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.366624:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574465.366639:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574465.366654:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495620736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574465.366679:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff93a249408e00 x1875453495620352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.366703:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495620352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.366717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.366721:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574465.366725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495620352:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574465.366751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574465.366777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574465.366791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff93a24940b800 x1875453495620480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.366803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495620480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.366812:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.366815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574465.366819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574465.366825:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620608:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574465.366849:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574465.366862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff93a249409c00 x1875453495620608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.366900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495620608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.366921:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.366925:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574465.366929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620608:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574465.366935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495620736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574465.366961:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574465.366973:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff93a24940a680 x1875453495620736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.366985:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495620736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.366993:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.366997:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574465.366999:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495620736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574466.326331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574466.326425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574467.350306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574467.350357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574468.374325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574468.374379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574469.398320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574469.398370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574470.422330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574470.422396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574470.486327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574470.486336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574470.486340:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574470.486348:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2686 00000100:00080000:0.0:1788574470.486356:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574470.486361:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574470.486395:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495620864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574470.486432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574470.486437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574470.486439:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574470.486443:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2686 00000100:00080000:0.0:1788574470.486448:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574470.486453:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574470.486462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574470.486466:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495620864:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574470.486475:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495620992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574470.486500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574470.486504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574470.486506:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574470.486510:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2686 00000100:00080000:0.0:1788574470.486513:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574470.486517:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574470.486521:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574470.486531:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495621120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574470.486537:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574470.486541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574470.486544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574470.486546:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574470.486549:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2686 00000100:00080000:0.0:1788574470.486558:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574470.486562:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574470.486568:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574470.486576:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff93a2460d0a80 x1875453495620864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574470.486578:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495621248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574470.486612:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495620864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.486623:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.486627:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574470.486631:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495620864:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574470.486654:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574470.486673:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574470.486683:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff93a2460d0700 x1875453495620992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.486691:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495620992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.486697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.486699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574470.486701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495620992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574470.486705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495621120:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574470.486716:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574470.486722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff93a2460d3b80 x1875453495621120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.486729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495621120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.486743:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.486745:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574470.486747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495621120:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574470.486751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495621248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574470.486765:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574470.486771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff93a2460d0000 x1875453495621248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.486778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495621248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.486783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.486785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574470.486787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495621248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574471.446335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574471.446376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574472.470343:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574472.470347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574473.494311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574473.494332:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574474.518338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574474.518377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574475.542339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574475.542364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574475.606331:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574475.606341:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574475.606345:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574475.606354:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2692 00000100:00080000:3.0:1788574475.606362:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574475.606367:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574475.606400:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495621376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574475.606435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574475.606438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574475.606440:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574475.606443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2692 00000100:00080000:3.0:1788574475.606445:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574475.606448:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574475.606456:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574475.606467:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495621504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574475.606476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574475.606478:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574475.606479:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574475.606481:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2692 00000100:00080000:3.0:1788574475.606483:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574475.606485:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574475.606490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574475.606495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495621376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574475.606500:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495621632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574475.606510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574475.606513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574475.606514:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574475.606516:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2692 00000100:00080000:3.0:1788574475.606524:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574475.606527:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574475.606532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574475.606559:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495621760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574475.606578:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574475.606634:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff93a25064df80 x1875453495621376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574475.606655:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495621376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574475.606667:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574475.606672:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574475.606676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495621376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574475.606709:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495621504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574475.606733:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574475.606747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff93a25064c380 x1875453495621504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574475.606761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495621504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574475.606771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574475.606775:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574475.606779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495621504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574475.606786:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495621632:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574475.606805:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574475.606816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff93a25064ca80 x1875453495621632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574475.606845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495621632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574475.606855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574475.606858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574475.606862:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495621632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574475.606869:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495621760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574475.606889:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574475.606900:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff93a25064c700 x1875453495621760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574475.606913:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495621760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574475.606922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574475.606926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574475.606929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495621760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574476.566313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574476.566365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574477.590303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574477.590371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574478.614350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574478.614417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574479.638308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574479.638398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574480.662343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574480.662388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574480.726348:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574480.726359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574480.726364:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574480.726373:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2697 00000100:00080000:3.0:1788574480.726384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574480.726392:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574480.726428:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495621888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574480.726457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574480.726462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574480.726463:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574480.726467:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2697 00000100:00080000:3.0:1788574480.726471:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574480.726475:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574480.726486:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574480.726500:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495622016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574480.726510:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495621888:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574480.726516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574480.726544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574480.726550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574480.726557:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2697 00000100:00080000:3.0:1788574480.726563:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574480.726566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788574480.726572:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574480.726582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574480.726612:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495622144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574480.726618:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff93a25064d500 x1875453495621888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.726632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574480.726637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574480.726639:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574480.726645:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495621888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.726646:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2697 00000100:00080000:3.0:1788574480.726654:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574480.726659:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574480.726669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788574480.726673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574480.726679:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574480.726683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495621888:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574480.726687:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495622272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574480.726708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574480.726729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574480.726740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff93a25064e300 x1875453495622016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574480.726748:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495622016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574480.726753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574480.726755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574480.726757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574480.726761:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622144:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574480.726772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574480.726778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff93a25064ce00 x1875453495622144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574480.726785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495622144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574480.726790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574480.726792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574480.726794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622144:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574480.726803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495622272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574480.726816:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574480.726822:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff93a25064f100 x1875453495622272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574480.726830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495622272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574480.726834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574480.726836:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574480.726838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495622272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574481.686362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574481.686426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574482.710343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574482.710436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574483.734292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574483.734389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574484.758331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574484.759373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574485.782334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574485.782404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574485.846363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.846374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574485.846379:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.846390:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2702 00000100:00080000:1.0:1788574485.846402:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.846410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574485.846455:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495622400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574485.846484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.846487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574485.846488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.846492:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2702 00000100:00080000:1.0:1788574485.846495:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.846518:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574485.846528:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574485.846543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495622528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574485.846548:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495622400:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574485.846555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.846564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574485.846566:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.846570:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2702 00000100:00080000:1.0:1788574485.846573:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.846575:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574485.846579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574485.846599:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495622656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574485.846609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.846614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574485.846616:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.846622:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2702 00000100:00080000:1.0:1788574485.846627:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.846632:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788574485.846641:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788574485.846644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574485.846685:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495622784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574485.846726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff93a258da1180 x1875453495622400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574485.846751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495622400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574485.846782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574485.846787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574485.846792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495622400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574485.846817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574485.846842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574485.846855:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff93a258da0380 x1875453495622528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574485.846869:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495622528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574485.846879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574485.846882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574485.846886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574485.846893:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622656:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574485.846911:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574485.846920:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff93a258da0000 x1875453495622656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574485.846933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495622656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574485.846941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574485.846954:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574485.846958:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495622656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574485.846970:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495622784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574485.846991:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574485.847002:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff93a258da2300 x1875453495622784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574485.847015:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495622784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574485.847024:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574485.847027:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574485.847031:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495622784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574486.806328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574486.806405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574487.830367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574487.830402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574488.854384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574488.854439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574489.878347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574489.878393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574490.902323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574490.902408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574490.966313:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.966340:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574490.966344:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.966354:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2707 00000100:00080000:1.0:1788574490.966364:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.966371:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574490.966407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495622912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.966437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.966445:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574490.966448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.966452:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2707 00000100:00080000:1.0:1788574490.966457:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.966462:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574490.966473:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574490.966487:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495622912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574490.966489:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495623040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.966502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.966505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574490.966506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.966510:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2707 00000100:00080000:1.0:1788574490.966513:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.966515:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574490.966520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574490.966530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495623168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.966542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.966544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574490.966546:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.966552:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2707 00000800:00000100:3.0:1788574490.966552:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574490.966561:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.966566:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574490.966573:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574490.966586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495623296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574490.966601:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff93a258da2300 x1875453495622912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574490.966618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495622912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574490.966627:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574490.966631:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574490.966633:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495622912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574490.966660:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574490.966676:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574490.966685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff93a258da0000 x1875453495623040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574490.966701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495623040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574490.966720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574490.966723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574490.966726:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574490.966730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623168:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574490.966747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574490.966755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff93a258da0380 x1875453495623168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574490.966763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495623168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574490.966768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574490.966770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574490.966772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623168:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574490.966775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495623296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574490.966784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574490.966790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff93a258da1180 x1875453495623296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574490.966797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495623296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574490.966811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574490.966814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574490.966816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495623296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574491.926302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574491.926356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574492.950309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574492.950391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574493.974303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574493.974339:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574494.998331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574494.998444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574496.022323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574496.022383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574496.086379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574496.086389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574496.086394:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574496.086404:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2712 00000100:00080000:2.0:1788574496.086414:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574496.086422:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574496.086457:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495623424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574496.086485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574496.086488:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574496.086490:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574496.086494:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2712 00000100:00080000:2.0:1788574496.086497:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574496.086501:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574496.086511:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574496.086524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495623552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574496.086534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574496.086537:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495623424:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574496.086538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574496.086541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574496.086549:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2712 00000100:00080000:2.0:1788574496.086553:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574496.086570:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574496.086575:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574496.086588:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495623680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574496.086599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574496.086603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574496.086605:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574496.086609:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2712 00000100:00080000:2.0:1788574496.086614:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574496.086616:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574496.086621:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574496.086626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574496.086638:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495623808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574496.086674:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff93a24940ad80 x1875453495623424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.086694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495623424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.086704:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.086709:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574496.086713:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495623424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574496.086770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574496.086797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574496.086812:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff93a249409180 x1875453495623552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.086825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495623552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.086850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.086853:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574496.086856:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574496.086860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623680:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574496.086891:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574496.086905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff93a249408a80 x1875453495623680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.086918:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495623680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.086927:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.086930:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574496.086932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495623680:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574496.086937:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495623808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574496.086952:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574496.086960:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff93a249408000 x1875453495623808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.086968:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495623808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.086973:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.086975:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574496.086982:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495623808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574497.046343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574497.046380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574498.070316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574498.070384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574499.094307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574499.094380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574500.118304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574500.118340:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574501.142347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574501.142402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574501.206398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574501.206412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574501.206417:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574501.206428:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2717 00000100:00080000:0.0:1788574501.206439:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574501.206447:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574501.206512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495623936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574501.206544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574501.206547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574501.206548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574501.206552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2717 00000100:00080000:0.0:1788574501.206556:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574501.206561:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574501.206574:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574501.206589:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495624064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574501.206594:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495623936:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574501.206608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574501.206612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574501.206615:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574501.206620:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2717 00000100:00080000:0.0:1788574501.206625:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574501.206629:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574501.206636:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574501.206650:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495624192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574501.206662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574501.206666:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574501.206667:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574501.206674:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574501.206682:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2717 00000100:00080000:0.0:1788574501.206688:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574501.206693:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574501.206698:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574501.206709:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495624320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574501.206732:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff93a2460d1500 x1875453495623936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.206750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495623936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.206759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.206763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574501.206781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495623936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574501.206807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574501.206827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574501.206837:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff93a2460d2680 x1875453495624064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.206846:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495624064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.206852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.206863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574501.206872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574501.206877:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624192:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574501.206894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574501.206903:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff93a2460d0380 x1875453495624192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.206911:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495624192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.206916:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.206918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574501.206921:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574501.206924:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495624320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574501.206935:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574501.206940:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff93a2460d3480 x1875453495624320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.206948:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495624320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.206953:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.206955:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574501.206957:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495624320:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574502.166344:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574502.166406:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574503.190366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574503.190451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574504.214362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574504.214372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574505.238312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574505.239336:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574506.262303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574506.262344:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574506.326349:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574506.326360:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574506.326365:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574506.326375:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2722 00000100:00080000:3.0:1788574506.326385:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574506.326393:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574506.326432:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495624448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574506.326462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574506.326465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574506.326467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574506.326470:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2722 00000100:00080000:3.0:1788574506.326477:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574506.326481:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574506.326490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574506.326504:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495624448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574506.326504:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495624576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574506.326518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574506.326521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574506.326522:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574506.326526:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2722 00000100:00080000:3.0:1788574506.326528:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574506.326530:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574506.326534:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574506.326555:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495624704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574506.326564:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788574506.326564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574506.326569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574506.326570:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574506.326574:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2722 00000100:00080000:3.0:1788574506.326579:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574506.326582:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574506.326586:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574506.326594:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495624832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574506.326605:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff93a25064d180 x1875453495624448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574506.326620:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495624448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574506.326629:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574506.326632:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574506.326636:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495624448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574506.326654:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574506.326669:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574506.326678:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff93a25064c000 x1875453495624576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574506.326693:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495624576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574506.326699:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574506.326701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574506.326703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574506.326707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624704:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574506.326720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574506.326727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff93a25064f480 x1875453495624704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574506.326734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495624704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574506.326748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574506.326750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574506.326753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495624704:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574506.326757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495624832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574506.326769:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574506.326776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff93a25064fb80 x1875453495624832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574506.326790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495624832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574506.326827:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574506.326830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574506.326833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495624832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574507.286332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574507.286378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574508.310329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574508.310357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574509.334352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574509.334380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574510.358353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574510.358444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574511.382344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574511.382398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574511.446326:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574511.446335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574511.446338:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574511.446347:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2727 00000100:00080000:1.0:1788574511.446355:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574511.446367:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574511.446399:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495624960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574511.446419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574511.446423:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574511.446425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574511.446430:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2727 00000100:00080000:1.0:1788574511.446433:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574511.446438:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574511.446448:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574511.446456:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495624960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574511.446460:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495625088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574511.446469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574511.446472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574511.446487:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574511.446491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2727 00000100:00080000:1.0:1788574511.446494:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574511.446498:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574511.446503:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574511.446513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495625216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574511.446515:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574511.446522:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574511.446525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574511.446527:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574511.446530:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2727 00000100:00080000:1.0:1788574511.446535:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574511.446539:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574511.446542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574511.446550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495625344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574511.446555:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff93a258da3480 x1875453495624960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.446572:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495624960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.446582:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.446586:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574511.446590:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495624960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574511.446613:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574511.446632:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574511.446649:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff93a258da2a00 x1875453495625088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.446658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495625088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.446667:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.446679:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574511.446683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574511.446690:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625216:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574511.446714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574511.446726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff93a258da3100 x1875453495625216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.446734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495625216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.446739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.446742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574511.446744:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574511.446747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495625344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574511.446768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574511.446776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff93a258da1880 x1875453495625344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.446783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495625344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.446793:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.446796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574511.446798:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495625344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574512.406359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574512.406423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574513.430274:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574513.430326:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574514.454337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574514.454394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574515.478349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574515.478420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574516.502313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574516.502395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574516.566315:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574516.566326:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574516.566330:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574516.566341:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2733 00000100:00080000:1.0:1788574516.566352:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574516.566369:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574516.566407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495625472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574516.566457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574516.566461:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574516.566464:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574516.566468:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2733 00000100:00080000:1.0:1788574516.566473:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574516.566478:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574516.566488:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574516.566503:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495625600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574516.566516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574516.566519:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495625472:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574516.566521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574516.566526:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574516.566534:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2733 00000100:00080000:1.0:1788574516.566538:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574516.566542:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574516.566547:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574516.566563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495625728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574516.566574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574516.566578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574516.566580:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574516.566584:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2733 00000800:00000100:3.0:1788574516.566589:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574516.566598:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574516.566604:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574516.566611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574516.566626:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495625856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574516.566634:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff93a258da1c00 x1875453495625472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.566652:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495625472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.566661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.566665:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574516.566668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495625472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574516.566688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574516.566704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574516.566712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff93a258da0700 x1875453495625600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.566720:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495625600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.566726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.566728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574516.566730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574516.566743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625728:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574516.566758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574516.566765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff93a258da0a80 x1875453495625728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.566772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495625728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.566777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.566779:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574516.566782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495625728:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574516.566785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495625856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574516.566795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574516.566801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff93a258da3800 x1875453495625856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.566808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495625856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.566822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.566826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574516.566829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495625856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574517.526332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574517.526382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574518.550324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574518.550469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574519.574370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574519.574520:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574520.598388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574520.598462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574521.622322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574521.622392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574521.686344:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574521.686352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574521.686355:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574521.686380:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2738 00000100:00080000:3.0:1788574521.686388:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574521.686394:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574521.686422:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495625984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574521.686445:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574521.686447:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574521.686449:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574521.686451:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2738 00000100:00080000:3.0:1788574521.686453:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574521.686455:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574521.686463:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574521.686474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495626112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574521.686486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574521.686488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495625984:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574521.686491:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574521.686496:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574521.686503:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2738 00000100:00080000:3.0:1788574521.686506:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574521.686510:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574521.686516:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574521.686530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495626240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574521.686557:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788574521.686565:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574521.686570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574521.686573:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574521.686578:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2738 00000100:00080000:3.0:1788574521.686582:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574521.686585:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574521.686589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574521.686601:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495626368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574521.686614:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff93a25064e680 x1875453495625984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574521.686637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495625984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574521.686649:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574521.686653:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574521.686657:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495625984:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574521.686682:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574521.686706:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574521.686718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff93a25064ed80 x1875453495626112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574521.686732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495626112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574521.686742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574521.686745:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574521.686757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574521.686765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626240:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574521.686788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574521.686800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff93a25064dc00 x1875453495626240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574521.686814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495626240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574521.686822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574521.686826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574521.686830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626240:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574521.686837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495626368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574521.686858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574521.686870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff93a25064d880 x1875453495626368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574521.686914:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495626368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574521.686925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574521.686929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574521.686932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495626368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574522.646337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574522.646393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574523.670313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574523.670361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574524.694318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574524.694360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574525.718310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574525.718387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574526.742400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574526.742471:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788574526.806332:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574526.806343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574526.806348:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574526.806359:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2743 00000100:00080000:3.0:1788574526.806369:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574526.806376:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574526.806428:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495626496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574526.806458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574526.806463:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574526.806465:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574526.806469:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2743 00000100:00080000:3.0:1788574526.806473:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574526.806494:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574526.806505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574526.806512:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495626496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574526.806540:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495626624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574526.806555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574526.806559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574526.806561:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574526.806566:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2743 00000100:00080000:3.0:1788574526.806572:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574526.806577:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574526.806582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574526.806596:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495626752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574526.806622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574526.806624:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626624:192.168.202.102@tcp:38:kworker.0 00000100:00080000:3.0:1788574526.806628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574526.806630:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574526.806640:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2743 00000100:00080000:3.0:1788574526.806645:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574526.806649:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574526.806656:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574526.806677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574526.806683:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495626880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574526.806702:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495626880:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788574527.766992:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574527.767673:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574528.790368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574528.791449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574529.814401:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574529.814567:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788574529.878489:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788574529.878563:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574529.878581:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574529.878625:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574529.878636:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574529.878644:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788574529.878736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574529] req@ffff93a25064ed80 x1875453495626880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574529.878763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495626880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574529.878783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574529.878789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574529.878793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495626880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574529.878799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574529] req@ffff93a25064dc00 x1875453495626752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574529.878809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495626752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574529.878815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574529.878817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574529.878819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574529.878823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574529] req@ffff93a25064d880 x1875453495626624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574529.878835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495626624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574529.878857:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574529.878862:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574529.878866:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495626624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574529.878873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574529] req@ffff93a25064ea00 x1875453495626496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574529.878886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495626496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574529.878894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574529.878898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574529.878901:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495626496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574530.838436:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574530.838558:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574531.862346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574531.862426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574531.926394:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574531.926407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574531.926412:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574531.926423:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2748 00000100:00080000:2.0:1788574531.926436:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574531.926444:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574531.926504:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495627008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574531.926542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574531.926585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574531.926588:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574531.926594:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2748 00000100:00080000:2.0:1788574531.926599:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574531.926605:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574531.926619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574531.926620:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495627008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574531.926641:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495627136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574531.926656:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574531.926661:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574531.926663:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574531.926668:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2748 00000100:00080000:2.0:1788574531.926673:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574531.926679:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574531.926686:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574531.926700:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495627264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574531.926713:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574531.926717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574531.926718:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574531.926725:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2748 00000800:00000100:1.0:1788574531.926727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574531.926731:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574531.926738:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574531.926753:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574531.926768:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495627392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574531.926804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff93a24940aa00 x1875453495627008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574531.926846:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495627008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574531.926861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574531.926866:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574531.926871:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495627008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574531.926909:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574531.926964:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574531.926980:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff93a249408700 x1875453495627136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574531.926996:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495627136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574531.927006:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574531.927011:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574531.927015:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574531.927023:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627264:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574531.927047:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574531.927059:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff93a249409880 x1875453495627264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574531.927073:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495627264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574531.927082:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574531.927086:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574531.927090:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627264:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574531.927097:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495627392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574531.927119:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574531.927131:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff93a249409500 x1875453495627392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574531.927161:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495627392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574531.927172:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574531.927175:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574531.927179:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495627392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574532.886338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574532.886351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788574533.910373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574533.910383:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574534.934347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574534.934355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574535.958314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574535.958395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574536.982365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574536.982378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574537.046346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.046361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574537.046366:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.046381:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2753 00000100:00080000:2.0:1788574537.046394:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.046403:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574537.046453:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495627520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.046488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.046492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574537.046494:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.046499:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2753 00000100:00080000:2.0:1788574537.046504:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.046511:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574537.046525:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574537.046543:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495627520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574537.046562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495627648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.046603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.046607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574537.046610:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.046617:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2753 00000100:00080000:2.0:1788574537.046623:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574537.046630:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574537.046636:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574537.046644:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574537.046656:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495627776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.046667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.046670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574537.046673:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.046681:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2753 00000100:00100000:1.0:1788574537.046682:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff93a24940b480 x1875453495627520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574537.046688:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.046698:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574537.046702:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574537.046720:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495627904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574537.046729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495627520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574537.046751:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574537.046758:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574537.046766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495627520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574537.046795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574537.046818:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574537.046828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff93a24940bb80 x1875453495627648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574537.046837:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495627648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574537.046859:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574537.046862:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574537.046864:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574537.046869:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627776:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574537.046885:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574537.046893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff93a24940b100 x1875453495627776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574537.046900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495627776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574537.046905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574537.046907:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574537.046909:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495627776:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574537.046913:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495627904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574537.046924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574537.046930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff93a249409500 x1875453495627904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574537.046937:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495627904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574537.046942:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574537.046943:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574537.046950:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495627904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574538.006327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574538.006383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574539.030323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574539.030380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574540.054295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574540.054336:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574541.078331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574541.078356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574542.102317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574542.102348:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574542.166306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.166315:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574542.166319:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.166327:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2758 00000100:00080000:2.0:1788574542.166337:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.166343:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574542.166376:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495628032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.166398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.166400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574542.166402:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.166404:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2758 00000100:00080000:2.0:1788574542.166407:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.166409:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574542.166417:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574542.166424:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495628160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.166431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.166434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574542.166436:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.166439:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2758 00000100:00080000:2.0:1788574542.166441:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.166444:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574542.166447:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574542.166455:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495628032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574542.166472:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495628288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.166481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.166483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574542.166485:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.166488:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2758 00000100:00080000:2.0:1788574542.166491:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.166493:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574542.166496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574542.166504:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495628416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574542.166511:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574542.166554:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff93a249409880 x1875453495628032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574542.166575:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495628032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574542.166586:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574542.166590:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574542.166594:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495628032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574542.166603:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574542.166625:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574542.166635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff93a249408700 x1875453495628160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574542.166655:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495628160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574542.166661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574542.166663:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574542.166666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574542.166684:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628288:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574542.166698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574542.166713:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff93a24940aa00 x1875453495628288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574542.166721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495628288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574542.166726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574542.166728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574542.166731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574542.166735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495628416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574542.166746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574542.166752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff93a249408000 x1875453495628416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574542.166760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495628416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574542.166765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574542.166767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574542.166769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495628416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574543.126355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574543.126365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574544.150356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574544.150414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574545.174303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574545.174416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574546.198345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574546.198413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574547.222316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574547.222425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574547.286337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574547.286347:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574547.286350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574547.286359:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2763 00000100:00080000:3.0:1788574547.286368:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574547.286374:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574547.286406:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495628544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574547.286429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574547.286432:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574547.286448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574547.286452:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2763 00000100:00080000:3.0:1788574547.286457:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574547.286461:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574547.286472:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574547.286486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495628672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574547.286497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574547.286502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574547.286503:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574547.286508:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2763 00000100:00080000:3.0:1788574547.286512:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574547.286519:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574547.286525:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574547.286538:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495628800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574547.286548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574547.286552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574547.286555:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574547.286559:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2763 00000100:00080000:3.0:1788574547.286563:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574547.286569:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574547.286577:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574547.286580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495628544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574547.286604:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495628928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574547.286689:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574547.286745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff93a25064e680 x1875453495628544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574547.286765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495628544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574547.286792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574547.286796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574547.286801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495628544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574547.286809:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574547.286833:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574547.286844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff93a25064fb80 x1875453495628672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574547.286854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495628672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574547.286861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574547.286864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574547.286868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574547.286886:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628800:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574547.286916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574547.286932:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff93a25064f480 x1875453495628800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574547.286948:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495628800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574547.286959:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574547.286964:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574547.286967:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495628800:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574547.287022:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495628928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574547.287064:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574547.287074:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff93a25064c000 x1875453495628928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574547.287089:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495628928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574547.287094:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574547.287097:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574547.287099:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495628928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574548.246323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574548.246377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574549.270325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574549.270433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574550.294339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574550.294434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574551.318308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574551.318384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574552.342356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574552.342433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574552.406415:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574552.406426:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574552.406431:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574552.406442:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2768 00000100:00080000:0.0:1788574552.406454:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574552.406461:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574552.406507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495629056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574552.406533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574552.406536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574552.406548:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574552.406553:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2768 00000100:00080000:0.0:1788574552.406557:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574552.406560:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574552.406571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574552.406581:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495629056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574552.406585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495629184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574552.406596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574552.406599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574552.406616:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574552.406620:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2768 00000100:00080000:0.0:1788574552.406624:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574552.406628:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574552.406633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574552.406642:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495629312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574552.406650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574552.406664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574552.406665:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574552.406668:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2768 00000100:00080000:0.0:1788574552.406671:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574552.406673:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574552.406676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574552.406683:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574552.406687:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495629440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574552.406735:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff93a2460d2300 x1875453495629056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574552.406753:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495629056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574552.406762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574552.406766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574552.406769:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495629056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574552.406794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574552.406812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574552.406821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff93a2460d1f80 x1875453495629184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574552.406838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495629184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574552.406845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574552.406847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574552.406849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574552.406853:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629312:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574552.406868:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574552.406875:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff93a2460d2a00 x1875453495629312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574552.406883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495629312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574552.406897:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574552.406898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574552.406900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629312:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574552.406904:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495629440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574552.406915:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574552.406921:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff93a2460d1180 x1875453495629440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574552.406928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495629440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574552.406943:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574552.406946:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574552.406960:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495629440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574553.366312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574553.366362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574554.390322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574554.390377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574555.414280:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574555.414337:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574556.438315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574556.438413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574557.462331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574557.462366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574557.526342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574557.526352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574557.526356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574557.526365:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2773 00000100:00080000:1.0:1788574557.526376:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574557.526384:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574557.526424:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495629568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574557.526450:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574557.526453:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574557.526455:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574557.526462:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2773 00000100:00080000:1.0:1788574557.526466:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574557.526469:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574557.526478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574557.526488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495629696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574557.526497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574557.526499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788574557.526509:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495629568:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574557.526516:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574557.526522:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2773 00000100:00080000:1.0:1788574557.526524:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574557.526527:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574557.526540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574557.526552:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495629824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574557.526560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574557.526566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574557.526568:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574557.526570:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2773 00000100:00080000:1.0:1788574557.526575:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574557.526575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574557.526582:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574557.526586:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574557.526596:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495629952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574557.526627:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff93a258da2680 x1875453495629568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.526642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495629568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.526649:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.526652:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574557.526655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495629568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574557.526661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574557.526683:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574557.526692:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff93a258da0e00 x1875453495629696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.526721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495629696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.526730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.526734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574557.526747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574557.526771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629824:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574557.526788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574557.526796:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff93a258da1f80 x1875453495629824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.526804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495629824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.526810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.526812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574557.526814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495629824:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574557.526817:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495629952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574557.526835:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574557.526842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff93a258da3800 x1875453495629952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.526849:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495629952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.526863:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.526865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574557.526868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495629952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574558.486321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574558.486363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574559.510315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574559.510473:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574560.534327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574560.534412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574561.558297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574561.558360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574562.582312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574562.582371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574562.646342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.646361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574562.646365:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.646376:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2779 00000100:00080000:2.0:1788574562.646386:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.646394:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574562.646426:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495630080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574562.646448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.646451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574562.646453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.646456:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2779 00000100:00080000:2.0:1788574562.646462:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.646465:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574562.646473:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574562.646482:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495630208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574562.646489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.646492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574562.646493:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.646496:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2779 00000100:00080000:2.0:1788574562.646498:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.646500:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574562.646504:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574562.646509:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495630080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574562.646529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495630336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574562.646538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.646540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574562.646542:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.646545:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2779 00000100:00080000:2.0:1788574562.646548:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.646550:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574562.646554:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574562.646561:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495630464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574562.646579:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574562.646623:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff93a249408a80 x1875453495630080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.646642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495630080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.646654:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.646658:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574562.646663:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495630080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574562.646671:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574562.646693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574562.646707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff93a249409180 x1875453495630208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.646721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495630208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.646738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.646743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574562.646747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574562.646772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630336:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574562.646796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574562.646809:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff93a24940ad80 x1875453495630336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.646821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495630336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.646830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.646833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574562.646837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630336:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574562.646843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495630464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574562.646864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574562.646877:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff93a24940a680 x1875453495630464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.646889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495630464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.646896:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.646898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574562.646908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495630464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574563.606301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574563.606353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574564.630302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574564.630350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574565.654331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574565.654376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574566.678332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574566.678409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574567.702339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574567.702431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574567.766293:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.766306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574567.766310:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.766322:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2784 00000100:00080000:2.0:1788574567.766335:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.766344:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574567.766398:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495630592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574567.766439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.766462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574567.766464:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.766469:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2784 00000100:00080000:2.0:1788574567.766474:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.766478:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574567.766490:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574567.766494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495630592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574567.766505:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495630720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574567.766515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.766518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574567.766519:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.766523:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2784 00000100:00080000:2.0:1788574567.766525:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.766528:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574567.766532:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574567.766540:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495630848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574567.766547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.766551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574567.766552:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.766555:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2784 00000100:00080000:2.0:1788574567.766557:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.766560:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574567.766569:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574567.766573:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574567.766583:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495630976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574567.766636:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff93a249409c00 x1875453495630592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.766654:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495630592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.766664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.766667:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574567.766670:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495630592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574567.766698:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574567.766715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574567.766724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff93a24940b800 x1875453495630720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.766737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495630720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.766746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.766749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574567.766763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574567.766770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630848:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574567.766797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574567.766810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff93a249408e00 x1875453495630848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.766825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495630848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.766835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.766838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574567.766842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495630848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574567.766849:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495630976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574567.766870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574567.766882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff93a24940a680 x1875453495630976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.766896:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495630976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.766905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.766908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574567.766912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495630976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574568.726320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574568.726387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574569.750323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574569.750384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574570.774302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574570.774379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574571.798295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574571.798374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574572.822338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574572.822438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574572.886310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.886322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574572.886326:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.886337:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2789 00000100:00080000:2.0:1788574572.886347:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.886353:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574572.886390:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495631104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574572.886426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.886429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574572.886431:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.886436:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2789 00000100:00080000:2.0:1788574572.886440:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.886446:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574572.886456:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574572.886473:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495631232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574572.886482:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495631104:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574572.886516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.886520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574572.886522:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.886526:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2789 00000100:00080000:2.0:1788574572.886547:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.886550:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574572.886557:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574572.886566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574572.886573:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495631360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574572.886588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.886593:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574572.886595:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.886600:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2789 00000100:00080000:2.0:1788574572.886605:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.886610:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574572.886615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574572.886624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff93a24940ad80 x1875453495631104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574572.886629:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495631488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574572.886682:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495631104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574572.886698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574572.886703:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574572.886707:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495631104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574572.886752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574572.886786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574572.886803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff93a249409180 x1875453495631232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574572.886818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495631232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574572.886829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574572.886833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574572.886837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574572.886844:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631360:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574572.886886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574572.886900:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff93a249408a80 x1875453495631360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574572.886916:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495631360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574572.886924:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574572.886927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574572.886931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631360:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574572.886939:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495631488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574572.886963:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574572.886977:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff93a249408000 x1875453495631488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574572.887006:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495631488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574572.887014:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574572.887018:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574572.887021:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495631488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574573.846331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574573.846412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574574.870342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574574.870428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574575.894333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574575.894392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574576.918300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574576.918363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574577.942312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574577.942361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574578.006354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574578.006365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574578.006370:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574578.006380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2794 00000100:00080000:2.0:1788574578.006391:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574578.006398:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574578.006433:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495631616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574578.006464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574578.006468:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574578.006470:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574578.006475:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2794 00000100:00080000:2.0:1788574578.006478:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574578.006482:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574578.006492:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574578.006505:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495631744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574578.006517:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495631616:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574578.006520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574578.006524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574578.006526:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574578.006531:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2794 00000100:00080000:2.0:1788574578.006534:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574578.006542:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574578.006548:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574578.006570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495631872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574578.006582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574578.006586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574578.006588:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574578.006596:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574578.006605:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2794 00000100:00080000:2.0:1788574578.006614:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574578.006620:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574578.006625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574578.006635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff93a24940aa00 x1875453495631616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574578.006641:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495632000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574578.006659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495631616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574578.006668:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574578.006670:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574578.006674:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495631616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574578.006694:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574578.006710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574578.006719:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff93a249408700 x1875453495631744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574578.006727:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495631744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574578.006733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574578.006735:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574578.006738:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574578.006741:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631872:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574578.006752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574578.006758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff93a249409880 x1875453495631872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574578.006776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495631872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574578.006790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574578.006793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574578.006795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495631872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574578.006799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495632000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574578.006814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574578.006821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff93a249409500 x1875453495632000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574578.006829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495632000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574578.006841:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574578.006843:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574578.006845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495632000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574578.966279:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574578.966351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574579.990299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574579.990364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574581.014327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574581.014399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574582.038337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574582.038391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574583.062363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574583.062458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574583.126369:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574583.126381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574583.126386:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574583.126398:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2799 00000100:00080000:1.0:1788574583.126410:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574583.126419:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574583.126474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495632128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574583.126526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574583.126530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574583.126531:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574583.126535:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2799 00000100:00080000:1.0:1788574583.126538:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574583.126542:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574583.126552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574583.126564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495632256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574583.126577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574583.126580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574583.126581:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574583.126585:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2799 00000100:00080000:1.0:1788574583.126587:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574583.126592:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574583.126595:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495632128:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574583.126597:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574583.126609:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495632384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574583.126617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574583.126620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574583.126622:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574583.126625:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2799 00000100:00080000:1.0:1788574583.126635:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574583.126639:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574583.126643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574583.126653:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495632512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574583.126675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574583.126723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff93a258da0a80 x1875453495632128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574583.126747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495632128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574583.126756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574583.126760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574583.126763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495632128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574583.126777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574583.126796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574583.126805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff93a258da0700 x1875453495632256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574583.126815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495632256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574583.126824:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574583.126828:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574583.126831:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574583.126863:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632384:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574583.126884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574583.126892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff93a258da1c00 x1875453495632384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574583.126900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495632384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574583.126935:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574583.126938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574583.126941:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632384:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574583.126947:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495632512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574583.126971:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574583.126984:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff93a258da1880 x1875453495632512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574583.127007:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495632512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574583.127013:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574583.127015:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574583.127017:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495632512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574584.086344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574584.086394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574585.110351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574585.110404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574586.134314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574586.134414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574587.158303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574587.158398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574588.182297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574588.182388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574588.246315:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574588.246325:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574588.246329:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574588.246338:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2804 00000100:00080000:0.0:1788574588.246346:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574588.246352:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574588.246398:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495632640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574588.246421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574588.246424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574588.246426:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574588.246429:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2804 00000100:00080000:0.0:1788574588.246432:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574588.246435:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574588.246443:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574588.246451:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495632768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574588.246457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574588.246460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574588.246461:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574588.246463:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2804 00000100:00080000:0.0:1788574588.246465:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574588.246467:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574588.246470:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574588.246476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495632896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574588.246483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574588.246486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574588.246494:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574588.246502:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2804 00000100:00080000:0.0:1788574588.246505:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574588.246510:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574588.246512:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495632640:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574588.246515:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574588.246528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495633024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574588.246574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574588.246609:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff93a2460d1180 x1875453495632640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574588.246632:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495632640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574588.246647:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574588.246650:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574588.246653:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495632640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574588.246658:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574588.246675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574588.246684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff93a2460d2a00 x1875453495632768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574588.246692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495632768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574588.246697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574588.246699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574588.246701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574588.246705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632896:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574588.246716:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574588.246722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff93a2460d1f80 x1875453495632896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574588.246729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495632896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574588.246734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574588.246736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574588.246738:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495632896:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574588.246764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495633024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574588.246777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574588.246783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff93a2460d2300 x1875453495633024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574588.246791:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495633024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574588.246796:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574588.246798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574588.246799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495633024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574589.206343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574589.206437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574590.230309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574590.230357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574591.254285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574591.254406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574592.278298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574592.278353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574593.302398:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574593.302872:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574593.366330:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574593.366340:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574593.366345:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574593.366356:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2809 00000100:00080000:0.0:1788574593.366366:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574593.366374:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574593.366416:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495633152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574593.366440:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574593.366443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574593.366445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574593.366448:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2809 00000100:00080000:0.0:1788574593.366451:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574593.366453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574593.366468:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574593.366483:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495633280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574593.366495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574593.366506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495633152:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574593.366517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574593.366519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574593.366523:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2809 00000100:00080000:0.0:1788574593.366527:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574593.366530:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574593.366535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574593.366549:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495633408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574593.366560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574593.366564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574593.366567:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574593.366571:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2809 00000100:00080000:0.0:1788574593.366575:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574593.366579:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574593.366585:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574593.366597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495633536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574593.366637:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574593.366675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574593.366706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495633536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574594.326505:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574594.326674:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574595.350552:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574595.351755:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574596.374331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574596.374658:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574596.438485:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574596.438520:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574596.438535:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574596.438566:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574596.438573:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574596.438579:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574596.438617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff93a2460d0380 x1875453495633280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574596.438658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495633280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574596.438671:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574596.438676:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574596.438681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574596.438689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff93a2460d2680 x1875453495633408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574596.438701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495633408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574596.438710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574596.438713:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574596.438716:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574596.438723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff93a2460d1500 x1875453495633536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574596.438736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495633536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574596.438743:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574596.438746:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574596.438749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495633536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574596.438756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff93a2460d3480 x1875453495633152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574596.438776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495633152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574596.438784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574596.438789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574596.438792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495633152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574597.398369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574597.398771:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574598.422391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574598.422472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574598.486423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574598.486431:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574598.486434:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574598.486443:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2814 00000100:00080000:2.0:1788574598.486451:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574598.486458:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574598.486487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495633664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574598.486513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574598.486515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574598.486517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574598.486520:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2814 00000100:00080000:2.0:1788574598.486522:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574598.486526:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574598.486534:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574598.486561:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495633664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574598.486562:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495633792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574598.486591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574598.486595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574598.486597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574598.486602:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2814 00000100:00080000:2.0:1788574598.486607:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574598.486613:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574598.486619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574598.486634:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495633920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574598.486642:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574598.486652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574598.486657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574598.486659:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574598.486665:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2814 00000100:00080000:2.0:1788574598.486669:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574598.486673:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574598.486679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574598.486693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff93a24940b100 x1875453495633664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574598.486695:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495634048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574598.486749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495633664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574598.486779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574598.486785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574598.486790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495633664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574598.486821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574598.486875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574598.486893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff93a24940bb80 x1875453495633792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574598.486905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495633792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574598.486923:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574598.486927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574598.486931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574598.486938:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633920:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574598.486964:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574598.486977:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff93a24940b480 x1875453495633920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574598.486988:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495633920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574598.486998:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574598.487002:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574598.487005:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495633920:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574598.487014:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495634048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574598.487036:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574598.487048:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff93a249409500 x1875453495634048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574598.487061:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495634048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574598.487070:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574598.487074:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574598.487086:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495634048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574599.446303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574599.446414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574600.470330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574600.470400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574601.494334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574601.494395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574602.518287:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574602.518378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574603.542310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574603.542329:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574603.606306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.606314:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574603.606318:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.606326:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2820 00000100:00080000:0.0:1788574603.606335:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.606353:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574603.606384:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495634176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574603.606412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.606415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574603.606416:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.606419:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2820 00000100:00080000:0.0:1788574603.606422:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.606425:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574603.606434:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574603.606444:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495634304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574603.606451:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495634176:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574603.606456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.606464:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574603.606466:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.606470:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2820 00000100:00080000:0.0:1788574603.606475:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.606478:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574603.606482:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574603.606491:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495634432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574603.606500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.606502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1788574603.606510:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574603.606511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.606519:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2820 00000100:00080000:0.0:1788574603.606526:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.606531:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574603.606537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574603.606545:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff93a2460d0000 x1875453495634176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574603.606553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495634560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574603.606564:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495634176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.606574:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.606579:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574603.606584:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495634176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574603.606610:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574603.606635:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574603.606645:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff93a2460d3b80 x1875453495634304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.606654:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495634304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.606660:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.606662:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574603.606670:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574603.606675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634432:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574603.606704:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574603.606712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff93a2460d0700 x1875453495634432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.606720:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495634432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.606725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.606727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574603.606730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634432:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574603.606734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495634560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574603.606744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574603.606750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff93a2460d0a80 x1875453495634560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.606757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495634560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.606764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.606767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574603.606770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495634560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574604.566285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574604.566335:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574605.590334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574605.590346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574606.614307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574606.614396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574607.638285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574607.638327:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574608.662284:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574608.662335:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574608.726431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574608.726440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574608.726443:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574608.726452:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2825 00000100:00080000:0.0:1788574608.726461:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574608.726467:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574608.726501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495634688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574608.726525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574608.726528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574608.726529:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574608.726532:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2825 00000100:00080000:0.0:1788574608.726534:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574608.726537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574608.726544:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574608.726553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495634816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574608.726572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495634688:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574608.726599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574608.726602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574608.726603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574608.726607:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2825 00000100:00080000:0.0:1788574608.726611:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574608.726614:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574608.726619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574608.726630:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495634944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574608.726636:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574608.726646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574608.726649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574608.726650:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574608.726655:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2825 00000100:00080000:0.0:1788574608.726659:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574608.726661:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574608.726664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574608.726671:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495635072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574608.726680:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff93a2460d0a80 x1875453495634688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.726696:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495634688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.726705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.726709:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574608.726713:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495634688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574608.726732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574608.726747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574608.726755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff93a2460d0700 x1875453495634816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.726775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495634816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.726782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.726784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574608.726787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574608.726791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634944:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574608.726822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574608.726831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff93a2460d3b80 x1875453495634944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.726839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495634944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.726844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.726846:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574608.726848:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495634944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574608.726852:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495635072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574608.726862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574608.726868:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff93a2460d0000 x1875453495635072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.726875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495635072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.726880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.726882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574608.726884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495635072:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788574609.686314:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574609.686314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574610.710295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574610.710338:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574611.734300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574611.734329:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574612.758300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574612.758329:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574613.782305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574613.782326:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574613.846322:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574613.846329:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574613.846333:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574613.846341:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2830 00000100:00080000:1.0:1788574613.846350:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574613.846356:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574613.846396:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495635200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574613.846418:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574613.846421:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574613.846435:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574613.846438:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2830 00000100:00080000:1.0:1788574613.846441:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574613.846444:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574613.846452:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574613.846461:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495635328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574613.846469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574613.846471:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574613.846472:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574613.846476:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2830 00000100:00080000:1.0:1788574613.846478:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574613.846480:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574613.846484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574613.846492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495635456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574613.846501:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495635200:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574613.846502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574613.846505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574613.846506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574613.846509:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2830 00000100:00080000:1.0:1788574613.846511:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574613.846516:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574613.846522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574613.846537:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495635584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574613.846582:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574613.846632:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff93a258da3100 x1875453495635200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574613.846654:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495635200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574613.846665:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574613.846669:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574613.846673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495635200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574613.846681:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574613.846705:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574613.846719:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff93a258da2a00 x1875453495635328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574613.846746:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495635328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574613.846756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574613.846760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574613.846764:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574613.846787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635456:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574613.846804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574613.846812:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff93a258da3480 x1875453495635456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574613.846831:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495635456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574613.846836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574613.846838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574613.846840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635456:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574613.846844:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495635584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574613.846856:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574613.846862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff93a258da1180 x1875453495635584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574613.846878:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495635584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574613.846887:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574613.846890:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574613.846893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495635584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574614.806333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574614.806365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574615.830305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574615.830325:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574616.854322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574616.854346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574617.878343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574617.878364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574618.902321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574618.902389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574618.966351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574618.966362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574618.966366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574618.966376:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2835 00000100:00080000:1.0:1788574618.966388:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574618.966395:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574618.966433:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495635712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574618.966464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574618.966468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574618.966470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574618.966475:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2835 00000100:00080000:1.0:1788574618.966479:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574618.966484:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574618.966493:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574618.966506:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495635840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574618.966520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574618.966539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574618.966541:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574618.966547:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2835 00000100:00080000:1.0:1788574618.966554:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574618.966562:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574618.966567:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495635712:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574618.966571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574618.966592:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495635968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574618.966609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574618.966614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574618.966618:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574618.966623:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2835 00000100:00080000:1.0:1788574618.966628:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574618.966633:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574618.966639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574618.966656:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495636096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574618.966662:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574618.966758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff93a258da0380 x1875453495635712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574618.966792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495635712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574618.966803:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574618.966807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574618.966811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495635712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574618.966818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574618.966839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574618.966850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff93a258da0000 x1875453495635840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574618.966858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495635840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574618.966864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574618.966867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574618.966869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574618.966894:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635968:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574618.966907:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574618.966914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff93a258da2300 x1875453495635968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574618.966922:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495635968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574618.966938:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574618.966941:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574618.966943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495635968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574618.966948:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495636096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574618.966962:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574618.966969:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff93a258da1180 x1875453495636096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574618.966976:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495636096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574618.966982:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574618.966983:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574618.966985:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495636096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574619.926285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574619.926321:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574620.950284:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574620.950347:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574621.974331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574621.974351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574622.998300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574622.998320:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574624.022376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574624.022407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574624.086339:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.086351:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574624.086378:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.086390:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2840 00000100:00080000:2.0:1788574624.086401:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.086409:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574624.086449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495636224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574624.086476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.086479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574624.086480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.086484:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2840 00000100:00080000:2.0:1788574624.086486:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.086489:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574624.086499:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574624.086508:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495636352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574624.086515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.086518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574624.086519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.086522:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2840 00000100:00080000:2.0:1788574624.086524:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.086526:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574624.086529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574624.086538:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495636480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574624.086538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495636224:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574624.086564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.086567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574624.086568:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.086572:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2840 00000100:00080000:2.0:1788574624.086575:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.086577:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574624.086580:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574624.086590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495636608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574624.086628:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574624.086706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff93a249409880 x1875453495636224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.086725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495636224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.086741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.086745:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574624.086748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495636224:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574624.086755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574624.086776:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574624.086786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff93a249408700 x1875453495636352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.086794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495636352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.086800:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.086803:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574624.086805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574624.086827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636480:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574624.086839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574624.086854:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff93a24940aa00 x1875453495636480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.086862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495636480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.086868:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.086870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574624.086873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636480:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574624.086884:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495636608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574624.086899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574624.086906:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff93a249408000 x1875453495636608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.086914:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495636608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.086920:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.086922:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574624.086924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495636608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574625.046348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574625.046395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574626.070349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574626.070366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574627.094327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574627.094355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574628.118333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574628.118360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574629.142315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574629.142363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574629.206316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.206332:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574629.206335:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.206343:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2845 00000100:00080000:2.0:1788574629.206352:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.206358:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574629.206399:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495636736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.206434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.206438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574629.206440:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.206445:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2845 00000100:00080000:2.0:1788574629.206451:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.206457:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574629.206476:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495636736:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574629.206485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574629.206506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495636864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.206521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.206526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574629.206528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.206534:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2845 00000800:00000100:3.0:1788574629.206539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574629.206540:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.206549:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574629.206556:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574629.206571:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495636992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.206582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.206586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574629.206588:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.206593:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2845 00000100:00080000:2.0:1788574629.206598:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574629.206598:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff93a249408a80 x1875453495636736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574629.206623:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574629.206629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574629.206636:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495636736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574629.206643:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495637120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574629.206648:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.206656:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574629.206666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495636736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574629.206687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574629.206711:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574629.206725:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff93a249409180 x1875453495636864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.206740:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495636864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.206749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.206752:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574629.206754:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574629.206759:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636992:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574629.206775:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574629.206783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff93a24940ad80 x1875453495636992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.206790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495636992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.206795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.206797:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574629.206799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495636992:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574629.206803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495637120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574629.206813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574629.206826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff93a24940a680 x1875453495637120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.206834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495637120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.206839:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.206841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574629.206843:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495637120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574630.166373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574630.166425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574631.190435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574631.190465:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574632.214318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574632.214384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574633.238286:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574633.238378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574634.262369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574634.262462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574634.326340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574634.326349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574634.326369:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574634.326379:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2850 00000100:00080000:2.0:1788574634.326390:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574634.326397:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574634.326433:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495637248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574634.326458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574634.326461:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574634.326463:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574634.326466:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2850 00000100:00080000:2.0:1788574634.326469:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574634.326472:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574634.326479:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574634.326490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495637376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574634.326497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574634.326500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574634.326501:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574634.326504:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2850 00000100:00080000:2.0:1788574634.326506:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574634.326508:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574634.326512:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574634.326518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495637248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574634.326525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495637504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574634.326566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574634.326571:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574634.326574:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574634.326578:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2850 00000100:00080000:2.0:1788574634.326584:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574634.326588:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574634.326595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574634.326595:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574634.326611:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495637632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574634.326638:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff93a249408e00 x1875453495637248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574634.326653:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495637248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574634.326661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574634.326663:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574634.326667:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495637248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574634.326672:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495637376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574634.326692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574634.326715:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff93a24940b800 x1875453495637376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574634.326729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495637376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574634.326739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574634.326743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574634.326745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495637376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574634.326764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495637504:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574634.326781:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574634.326789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff93a249409c00 x1875453495637504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574634.326797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495637504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574634.326802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574634.326805:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574634.326807:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495637504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574634.326810:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495637632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574634.326821:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574634.326827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff93a24940a680 x1875453495637632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574634.326834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495637632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574634.326844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574634.326847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574634.326849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495637632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574635.286303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574635.286345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574636.310291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574636.310335:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574637.334291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574637.334336:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574638.358314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574638.358371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574639.382294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574639.382372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574639.446328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574639.446336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574639.446339:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574639.446347:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2855 00000100:00080000:0.0:1788574639.446356:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574639.446362:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574639.446394:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495637760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574639.446415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574639.446418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574639.446419:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574639.446422:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2855 00000100:00080000:0.0:1788574639.446424:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574639.446426:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574639.446434:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574639.446443:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495637888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574639.446451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574639.446454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574639.446455:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574639.446472:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495637760:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574639.446474:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2855 00000100:00080000:0.0:1788574639.446482:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574639.446486:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574639.446490:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574639.446516:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495638016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574639.446525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574639.446527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574639.446529:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574639.446531:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2855 00000100:00080000:0.0:1788574639.446534:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574639.446537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574639.446541:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574639.446550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495638144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574639.446578:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574639.446621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff93a2460d3480 x1875453495637760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574639.446641:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495637760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574639.446652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574639.446656:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574639.446660:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495637760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574639.446667:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495637888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574639.446692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574639.446706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff93a2460d1500 x1875453495637888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574639.446719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495637888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574639.446738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574639.446742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574639.446746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495637888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574639.446771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638016:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574639.446796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574639.446807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff93a2460d2680 x1875453495638016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574639.446819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495638016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574639.446827:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574639.446830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574639.446834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638016:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574639.446841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495638144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574639.446861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574639.446872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff93a2460d0380 x1875453495638144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574639.446885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495638144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574639.446893:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574639.446896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574639.446918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495638144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574640.406305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574640.406369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574641.430331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574641.430419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574642.454304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574642.454345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574643.478388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574643.478416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574644.502297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574644.502377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574644.566299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574644.566310:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574644.566315:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574644.566326:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2861 00000100:00080000:0.0:1788574644.566336:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574644.566361:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574644.566390:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495638272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574644.566412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574644.566415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574644.566417:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574644.566420:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2861 00000100:00080000:0.0:1788574644.566423:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574644.566425:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574644.566432:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574644.566441:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495638400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574644.566473:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574644.566476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574644.566477:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574644.566483:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2861 00000100:00100000:3.0:1788574644.566484:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495638272:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574644.566485:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574644.566489:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574644.566493:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574644.566505:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495638528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574644.566513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574644.566515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574644.566522:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574644.566525:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2861 00000100:00080000:0.0:1788574644.566529:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574644.566532:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574644.566536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574644.566544:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495638656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574644.566562:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574644.566605:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff93a2460d2300 x1875453495638272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574644.566624:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495638272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574644.566634:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574644.566637:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574644.566641:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495638272:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574644.566650:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574644.566672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574644.566683:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff93a2460d1f80 x1875453495638400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574644.566694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495638400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574644.566702:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574644.566705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574644.566715:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574644.566739:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638528:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574644.566759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574644.566769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff93a2460d2a00 x1875453495638528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574644.566781:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495638528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574644.566790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574644.566793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574644.566796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574644.566802:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495638656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574644.566818:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574644.566827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff93a2460d1180 x1875453495638656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574644.566838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495638656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574644.566847:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574644.566850:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574644.566854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495638656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574645.526339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574645.526395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574646.550346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574646.550399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574647.574285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574647.574358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574648.598299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574648.598416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574649.622313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574649.622416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574649.686389:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574649.686408:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574649.686412:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574649.686424:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2866 00000100:00080000:2.0:1788574649.686433:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574649.686440:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574649.686471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495638784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574649.686496:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574649.686499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574649.686500:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574649.686503:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2866 00000100:00080000:2.0:1788574649.686505:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574649.686508:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574649.686529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574649.686538:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495638912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574649.686548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574649.686550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574649.686552:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574649.686557:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2866 00000100:00100000:3.0:1788574649.686559:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495638784:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574649.686560:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574649.686566:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574649.686573:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574649.686581:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495639040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574649.686589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574649.686592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574649.686593:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574649.686596:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2866 00000100:00080000:2.0:1788574649.686598:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574649.686601:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574649.686604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574649.686611:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495639168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574649.686629:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574649.686666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff93a24940ad80 x1875453495638784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574649.686680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495638784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574649.686688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574649.686691:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574649.686694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495638784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574649.686712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574649.686726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574649.686734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff93a249409180 x1875453495638912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574649.686742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495638912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574649.686748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574649.686750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574649.686752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495638912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574649.686756:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639040:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574649.686780:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574649.686787:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff93a249408a80 x1875453495639040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574649.686794:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495639040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574649.686800:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574649.686802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574649.686805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639040:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574649.686808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495639168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574649.686817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574649.686823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff93a249408000 x1875453495639168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574649.686830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495639168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574649.686835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574649.686837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574649.686839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495639168:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788574650.646379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574650.646434:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574651.670364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574651.670453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574652.694330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574652.694397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574653.718334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574653.718449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574654.742310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574654.742385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574654.806376:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.806387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574654.806392:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.806404:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2871 00000100:00080000:1.0:1788574654.806417:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.806424:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574654.806478:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495639296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574654.806514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.806518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574654.806521:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.806526:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2871 00000100:00080000:1.0:1788574654.806531:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.806537:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574654.806551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574654.806564:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495639296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574654.806567:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495639424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574654.806575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.806578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574654.806580:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.806585:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2871 00000100:00080000:1.0:1788574654.806590:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.806593:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574654.806606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574654.806615:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495639552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574654.806624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.806627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574654.806629:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.806632:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2871 00000100:00080000:1.0:1788574654.806635:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.806647:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574654.806651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574654.806659:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495639680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574654.806668:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574654.806715:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff93a258da3480 x1875453495639296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574654.806732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495639296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574654.806741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574654.806749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574654.806752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495639296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574654.806778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574654.806795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574654.806804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff93a258da2a00 x1875453495639424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574654.806813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495639424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574654.806818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574654.806821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574654.806823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574654.806826:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639552:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574654.806839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574654.806844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff93a258da3100 x1875453495639552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574654.806862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495639552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574654.806868:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574654.806870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574654.806872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639552:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574654.806876:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495639680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574654.806888:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574654.806894:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff93a258da1880 x1875453495639680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574654.806901:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495639680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574654.806907:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574654.806910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574654.806913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495639680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574655.766333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574655.766423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574656.790289:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574656.790391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574657.814299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574657.814370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574658.838533:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574658.838537:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574659.862725:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574659.863117:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574659.926360:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574659.926375:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574659.926381:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574659.926395:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2876 00000100:00080000:1.0:1788574659.926409:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574659.926419:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574659.926466:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495639808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574659.926503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574659.926506:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574659.926525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574659.926529:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2876 00000100:00080000:1.0:1788574659.926534:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574659.926538:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574659.926550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574659.926560:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495639808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574659.926564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495639936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574659.926575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574659.926577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574659.926579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574659.926582:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2876 00000100:00080000:1.0:1788574659.926585:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574659.926587:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574659.926591:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574659.926598:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495640064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574659.926606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574659.926609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574659.926611:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574659.926613:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2876 00000100:00080000:1.0:1788574659.926616:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574659.926618:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574659.926620:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574659.926634:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495640192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574659.926690:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574659.926713:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574659.926748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495640192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574660.886951:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574660.887112:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788574661.911663:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574661.911722:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574662.934638:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574662.934995:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:2.0:1788574662.998531:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574662.998594:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574662.998612:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574662.998653:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574662.998669:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574662.998679:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574662.999047:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574662] req@ffff93a258da0700 x1875453495639936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574662.999076:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495639936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574662.999090:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574662.999096:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574662.999101:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495639936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574662.999111:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574662] req@ffff93a258da0a80 x1875453495640064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574662.999121:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495640064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574662.999129:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574662.999133:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574662.999137:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574662.999209:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574662] req@ffff93a258da3800 x1875453495640192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574662.999226:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495640192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574662.999254:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574662.999283:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574662.999287:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495640192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574662.999296:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574662] req@ffff93a258da1c00 x1875453495639808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574662.999312:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495639808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574662.999321:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574662.999326:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574662.999332:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495639808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574663.958316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574663.958640:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574664.982337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574664.982375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574665.046334:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574665.046345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574665.046350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574665.046361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2881 00000100:00080000:3.0:1788574665.046372:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574665.046379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574665.046413:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495640320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574665.046443:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574665.046447:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574665.046449:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574665.046455:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2881 00000100:00080000:3.0:1788574665.046464:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574665.046485:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495640320:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574665.046486:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574665.046504:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574665.046522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495640448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574665.046534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574665.046539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574665.046541:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574665.046547:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2881 00000100:00080000:3.0:1788574665.046552:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574665.046557:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574665.046562:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574665.046575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574665.046577:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495640576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574665.046591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574665.046596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574665.046598:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574665.046603:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2881 00000100:00080000:3.0:1788574665.046607:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574665.046612:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574665.046619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574665.046628:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff93a25064d180 x1875453495640320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574665.046634:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495640704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574665.046666:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495640320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574665.046679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574665.046684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574665.046689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495640320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574665.046715:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574665.046742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574665.046756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff93a25064f100 x1875453495640448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574665.046769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495640448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574665.046778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574665.046782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574665.046785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574665.046791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640576:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574665.046827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574665.046842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff93a25064ce00 x1875453495640576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574665.046853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495640576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574665.046862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574665.046865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574665.046868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640576:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574665.046875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495640704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574665.046897:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574665.046908:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff93a25064e300 x1875453495640704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574665.046920:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495640704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574665.046928:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574665.046931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574665.046934:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495640704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574666.006310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574666.006356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574667.030359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574667.030452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574668.054318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574668.054490:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574669.078327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574669.078426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574670.102347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574670.102392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574670.166316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.166339:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574670.166343:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.166353:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2886 00000100:00080000:3.0:1788574670.166364:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.166372:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574670.166408:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495640832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.166439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.166442:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574670.166444:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.166447:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2886 00000100:00080000:3.0:1788574670.166451:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.166454:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574670.166461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574670.166473:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495640960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.166480:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.166483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574670.166484:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.166487:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2886 00000100:00080000:3.0:1788574670.166490:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574670.166495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495640832:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574670.166495:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574670.166500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574670.166510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495641088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.166524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.166526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574670.166528:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.166531:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2886 00000100:00080000:3.0:1788574670.166534:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.166536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574670.166540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574670.166548:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495641216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574670.166567:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574670.166610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff93a25064d500 x1875453495640832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574670.166643:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495640832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574670.166654:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574670.166658:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574670.166662:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495640832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574670.166670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574670.166692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574670.166714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff93a25064c700 x1875453495640960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574670.166731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495640960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574670.166740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574670.166743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574670.166746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495640960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574670.166773:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641088:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574670.166794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574670.166805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff93a25064ca80 x1875453495641088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574670.166817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495641088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574670.166837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574670.166841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574670.166845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641088:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574670.166852:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495641216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574670.166872:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574670.166882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff93a25064c380 x1875453495641216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574670.166894:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495641216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574670.166902:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574670.166920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574670.166924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495641216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574671.126314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574671.126435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574672.150318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574672.150412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574673.174318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574673.174372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574674.198331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574674.198395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574675.222377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574675.222410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574675.286371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574675.286383:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574675.286388:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574675.286397:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2891 00000100:00080000:0.0:1788574675.286405:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574675.286410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574675.286446:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495641344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574675.286469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574675.286472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574675.286474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574675.286478:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2891 00000100:00080000:0.0:1788574675.286481:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574675.286484:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574675.286492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574675.286500:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495641344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574675.286505:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495641472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574675.286523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574675.286525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574675.286539:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574675.286543:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2891 00000100:00080000:0.0:1788574675.286562:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574675.286565:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574675.286570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574675.286575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574675.286585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495641600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574675.286598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574675.286602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574675.286604:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574675.286609:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2891 00000100:00080000:0.0:1788574675.286614:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574675.286619:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574675.286623:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff93a2460d1180 x1875453495641344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574675.286627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574675.286650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495641344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574675.286655:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495641728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574675.286667:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574675.286675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574675.286680:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495641344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574675.286707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574675.286726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574675.286735:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff93a2460d2a00 x1875453495641472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574675.286744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495641472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574675.286750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574675.286753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574675.286755:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574675.286758:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641600:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574675.286769:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574675.286775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff93a2460d1f80 x1875453495641600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574675.286783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495641600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574675.286788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574675.286790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574675.286792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574675.286795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495641728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574675.286805:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574675.286810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff93a2460d2300 x1875453495641728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574675.286823:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495641728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574675.286829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574675.286831:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574675.286833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495641728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574676.246396:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574676.246550:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574677.270323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574677.270375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574678.294345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574678.294401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574679.318336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574679.318389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574680.342301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574680.342442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574680.406334:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.406342:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574680.406346:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.406354:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2896 00000100:00080000:0.0:1788574680.406364:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.406370:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574680.406401:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495641856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.406441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.406443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574680.406445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.406448:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2896 00000100:00080000:0.0:1788574680.406451:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.406453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574680.406461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574680.406471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495641984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.406478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.406480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574680.406481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.406484:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2896 00000100:00080000:0.0:1788574680.406486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.406488:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574680.406491:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574680.406494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495641856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574680.406501:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495642112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.406514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.406518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574680.406520:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.406524:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2896 00000100:00080000:0.0:1788574680.406547:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.406553:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788574680.406556:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788574680.406563:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574680.406580:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495642240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574680.406608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff93a2460d0380 x1875453495641856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574680.406627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495641856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574680.406636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574680.406639:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574680.406643:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495641856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574680.406648:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574680.406665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574680.406673:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff93a2460d2680 x1875453495641984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574680.406681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495641984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574680.406698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574680.406700:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574680.406703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495641984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574680.406722:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495642112:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574680.406737:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574680.406744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff93a2460d1500 x1875453495642112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574680.406751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495642112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574680.406757:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574680.406759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574680.406761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495642112:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574680.406764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495642240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574680.406773:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574680.406779:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff93a2460d3480 x1875453495642240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574680.406786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495642240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574680.406791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574680.406793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574680.406800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495642240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574681.366327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574681.366494:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574682.390313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574682.390429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574683.414383:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574683.414458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574684.438297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574684.438405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574685.462289:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574685.462371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574685.526330:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574685.526340:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574685.526343:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574685.526352:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2901 00000100:00080000:1.0:1788574685.526361:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574685.526367:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574685.526398:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495642368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574685.526425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574685.526429:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574685.526431:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574685.526435:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2901 00000100:00080000:1.0:1788574685.526439:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574685.526443:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574685.526454:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574685.526470:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495642496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574685.526474:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495642368:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574685.526487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574685.526490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574685.526492:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574685.526497:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2901 00000100:00080000:1.0:1788574685.526500:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574685.526520:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574685.526527:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574685.526539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574685.526544:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495642624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574685.526558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574685.526562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574685.526563:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574685.526567:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2901 00000100:00080000:1.0:1788574685.526573:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574685.526576:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574685.526579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574685.526597:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff93a258da1f80 x1875453495642368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574685.526599:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495642752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574685.526636:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495642368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.526647:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.526651:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574685.526654:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495642368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574685.526673:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495642496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574685.526701:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574685.526710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff93a258da0e00 x1875453495642496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.526718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495642496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.526724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.526726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574685.526728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495642496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574685.526732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495642624:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574685.526742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574685.526748:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff93a258da2680 x1875453495642624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.526755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495642624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.526770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.526773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574685.526775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495642624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574685.526780:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495642752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574685.526792:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574685.526799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff93a258da1c00 x1875453495642752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574685.526806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495642752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574685.526812:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574685.526813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574685.526815:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495642752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574686.486341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574686.486401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574687.510350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574687.510412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574688.534326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574688.534395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574689.558323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574689.558434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574690.582288:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574690.582353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574690.646367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.646379:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574690.646383:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.646411:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2907 00000100:00080000:2.0:1788574690.646422:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.646429:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574690.646485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495642880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574690.646516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.646520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574690.646523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.646527:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2907 00000100:00080000:2.0:1788574690.646532:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.646536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574690.646545:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574690.646557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495643008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574690.646566:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495642880:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574690.646568:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.646576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574690.646577:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.646581:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2907 00000100:00080000:2.0:1788574690.646584:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.646586:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574690.646590:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574690.646599:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495643136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574690.646616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.646620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574690.646621:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.646624:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2907 00000800:00000100:3.0:1788574690.646629:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574690.646630:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.646635:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574690.646639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574690.646653:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495643264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574690.646677:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff93a24940aa00 x1875453495642880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574690.646692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495642880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574690.646706:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574690.646709:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574690.646712:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495642880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574690.646730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574690.646748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574690.646756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff93a249408700 x1875453495643008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574690.646764:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495643008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574690.646770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574690.646772:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574690.646774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574690.646778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643136:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574690.646789:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574690.646794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff93a249409880 x1875453495643136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574690.646802:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495643136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574690.646807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574690.646808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574690.646810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643136:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574690.646819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495643264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574690.646833:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574690.646839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff93a249409500 x1875453495643264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574690.646848:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495643264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574690.646853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574690.646854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574690.646856:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495643264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574691.606313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574691.606368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574692.630365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574692.630462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574693.654331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574693.654412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574694.678354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574694.678446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574695.702346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574695.702427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574695.766334:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574695.766346:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574695.766351:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574695.766362:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2912 00000100:00080000:2.0:1788574695.766374:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574695.766382:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574695.766428:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495643392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574695.766456:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574695.766459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574695.766460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574695.766481:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2912 00000100:00080000:2.0:1788574695.766485:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574695.766490:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574695.766504:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574695.766513:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495643392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574695.766522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495643520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574695.766537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574695.766541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574695.766544:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574695.766549:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2912 00000100:00080000:2.0:1788574695.766553:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574695.766557:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574695.766562:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574695.766573:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495643648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574695.766582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574695.766585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574695.766586:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574695.766592:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2912 00000800:00000100:3.0:1788574695.766593:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574695.766598:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574695.766602:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574695.766606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574695.766615:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495643776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574695.766655:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff93a24940b480 x1875453495643392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574695.766673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495643392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574695.766683:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574695.766686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574695.766689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495643392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574695.766728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574695.766748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574695.766757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff93a24940bb80 x1875453495643520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574695.766766:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495643520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574695.766772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574695.766774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574695.766777:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574695.766782:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643648:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574695.766804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574695.766816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff93a24940b100 x1875453495643648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574695.766828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495643648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574695.766837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574695.766840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574695.766844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495643648:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574695.766850:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495643776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574695.766870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574695.766882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff93a249409500 x1875453495643776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574695.766906:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495643776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574695.766915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574695.766918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574695.766921:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495643776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574696.726299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574696.726408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574697.750347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574697.750389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574698.774297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574698.774355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574699.798301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574699.798383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574700.822295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574700.822362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574700.886314:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574700.886327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574700.886331:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574700.886340:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2917 00000100:00080000:2.0:1788574700.886349:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574700.886355:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574700.886387:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495643904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574700.886410:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574700.886412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574700.886414:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574700.886417:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2917 00000100:00080000:2.0:1788574700.886419:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574700.886421:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574700.886429:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574700.886438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495644032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574700.886445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574700.886447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574700.886448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574700.886451:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2917 00000100:00080000:2.0:1788574700.886453:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574700.886455:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574700.886462:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495643904:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574700.886474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574700.886485:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495644160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574700.886494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574700.886497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574700.886499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574700.886503:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2917 00000100:00080000:2.0:1788574700.886508:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574700.886511:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574700.886515:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574700.886524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495644288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574700.886524:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574700.886583:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff93a249409880 x1875453495643904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574700.886599:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495643904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574700.886607:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574700.886610:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574700.886613:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495643904:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574700.886618:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574700.886634:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574700.886642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff93a249408700 x1875453495644032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574700.886659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495644032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574700.886665:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574700.886673:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574700.886676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574700.886694:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644160:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574700.886710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574700.886727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff93a24940aa00 x1875453495644160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574700.886735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495644160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574700.886741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574700.886743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574700.886745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574700.886748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495644288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574700.886757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574700.886763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff93a249408000 x1875453495644288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574700.886770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495644288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574700.886784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574700.886787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574700.886789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495644288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574701.846388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574701.846502:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574702.870304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574702.870421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574703.894348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574703.894499:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574704.918354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574704.918415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574705.942292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574705.942349:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574706.006285:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574706.006295:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574706.006299:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574706.006308:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2922 00000100:00080000:2.0:1788574706.006318:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574706.006324:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574706.006357:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495644416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574706.006378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574706.006382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574706.006384:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574706.006388:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2922 00000100:00080000:2.0:1788574706.006407:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574706.006413:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574706.006423:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574706.006427:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495644416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574706.006436:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495644544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574706.006447:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574706.006450:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574706.006452:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574706.006456:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2922 00000100:00080000:2.0:1788574706.006459:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574706.006463:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574706.006467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574706.006476:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495644672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574706.006485:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574706.006486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574706.006490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574706.006492:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574706.006496:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2922 00000100:00080000:2.0:1788574706.006499:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574706.006503:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574706.006507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574706.006515:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495644800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574706.006531:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff93a249408a80 x1875453495644416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574706.006551:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495644416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574706.006571:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574706.006576:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574706.006581:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495644416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574706.006601:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574706.006621:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574706.006630:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff93a249409180 x1875453495644544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574706.006638:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495644544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574706.006653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574706.006657:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574706.006661:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574706.006667:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644672:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574706.006692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574706.006705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff93a24940ad80 x1875453495644672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574706.006714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495644672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574706.006721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574706.006723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574706.006725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495644672:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574706.006729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495644800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574706.006743:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574706.006750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff93a24940a680 x1875453495644800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574706.006767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495644800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574706.006772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574706.006774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574706.006782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495644800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574706.966301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574706.966364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574707.990333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574707.990392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574709.014357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574709.014402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574710.038354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574710.038476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574711.062338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574711.062409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574711.126380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574711.126393:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574711.126398:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574711.126409:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2927 00000100:00080000:1.0:1788574711.126422:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574711.126429:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574711.126473:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495644928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574711.126501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574711.126504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574711.126522:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574711.126526:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2927 00000100:00080000:1.0:1788574711.126529:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574711.126534:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574711.126547:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574711.126563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495645056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574711.126576:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495644928:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574711.126577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574711.126586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574711.126590:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574711.126595:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2927 00000100:00080000:1.0:1788574711.126598:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574711.126601:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574711.126622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574711.126635:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495645184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574711.126643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574711.126646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574711.126648:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574711.126651:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2927 00000100:00080000:1.0:1788574711.126654:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574711.126657:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574711.126660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574711.126674:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495645312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574711.126693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574711.126744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff93a258da3800 x1875453495644928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574711.126761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495644928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574711.126778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574711.126782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574711.126785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495644928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574711.126809:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574711.126829:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574711.126838:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff93a258da0a80 x1875453495645056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574711.126847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495645056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574711.126853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574711.126855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574711.126865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574711.126870:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645184:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574711.126886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574711.126893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff93a258da0700 x1875453495645184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574711.126901:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495645184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574711.126906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574711.126908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574711.126910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574711.126914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495645312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574711.126925:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574711.126931:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff93a258da1880 x1875453495645312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574711.126938:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495645312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574711.126943:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574711.126945:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574711.126947:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495645312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574712.086343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574712.086421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574713.110292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574713.110363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574714.134319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574714.134388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574715.158316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574715.158403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574716.182345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574716.182409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574716.246367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574716.246378:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574716.246384:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574716.246395:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2932 00000100:00080000:2.0:1788574716.246406:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574716.246414:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574716.246460:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495645440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574716.246488:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574716.246491:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574716.246492:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574716.246495:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2932 00000100:00080000:2.0:1788574716.246509:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574716.246512:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574716.246523:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574716.246534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495645568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574716.246541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574716.246544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574716.246545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574716.246548:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2932 00000100:00080000:2.0:1788574716.246550:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574716.246553:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574716.246559:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495645440:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574716.246588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574716.246601:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495645696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574716.246610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574716.246613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574716.246615:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574716.246618:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2932 00000100:00080000:2.0:1788574716.246632:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574716.246635:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574716.246638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574716.246649:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574716.246651:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495645824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574716.246704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff93a249409c00 x1875453495645440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574716.246722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495645440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574716.246731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574716.246734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574716.246737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495645440:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574716.246743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574716.246762:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574716.246771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff93a24940b800 x1875453495645568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574716.246779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495645568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574716.246785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574716.246787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574716.246789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574716.246813:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645696:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574716.246847:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574716.246860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff93a249408e00 x1875453495645696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574716.246874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495645696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574716.246880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574716.246882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574716.246885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495645696:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574716.246889:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495645824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574716.246904:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574716.246911:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff93a24940a680 x1875453495645824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574716.246918:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495645824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574716.246923:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574716.246925:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574716.246927:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495645824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574717.206296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574717.206376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574718.230308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574718.230403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574719.254298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574719.254385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574720.278358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574720.278443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574721.302299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574721.302362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574721.366364:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574721.366372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574721.366376:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574721.366384:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2937 00000100:00080000:1.0:1788574721.366393:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574721.366399:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574721.366441:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495645952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574721.366464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574721.366466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574721.366468:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574721.366471:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2937 00000100:00080000:1.0:1788574721.366474:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574721.366476:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574721.366484:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574721.366492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495646080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574721.366499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574721.366503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574721.366505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574721.366507:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2937 00000100:00080000:1.0:1788574721.366510:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574721.366525:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574721.366530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574721.366539:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495645952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574721.366541:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495646208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574721.366550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574721.366553:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574721.366554:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574721.366570:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2937 00000100:00080000:1.0:1788574721.366574:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574721.366576:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574721.366580:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574721.366589:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495646336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574721.366645:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574721.366711:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff93a258da3100 x1875453495645952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574721.366730:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495645952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574721.366739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574721.366743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574721.366746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495645952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574721.366754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574721.366780:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574721.366795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff93a258da2a00 x1875453495646080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574721.366809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495646080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574721.366818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574721.366821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574721.366824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574721.366853:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646208:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574721.366873:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574721.366881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff93a258da3480 x1875453495646208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574721.366900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495646208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574721.366906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574721.366908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574721.366910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646208:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574721.366914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495646336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574721.366928:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574721.366936:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff93a258da1180 x1875453495646336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574721.366944:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495646336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574721.366949:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574721.366951:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574721.366953:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495646336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574722.326304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574722.326398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574723.350327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574723.350403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574724.374328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574724.374384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574725.398292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574725.398372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574726.422613:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574726.423294:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574726.486316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.486328:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574726.486332:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.486343:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2942 00000100:00080000:0.0:1788574726.486352:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.486358:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574726.486391:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495646464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574726.486414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.486417:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574726.486419:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.486422:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2942 00000100:00080000:0.0:1788574726.486424:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.486426:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574726.486435:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574726.486444:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495646592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574726.486452:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.486455:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574726.486469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.486480:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2942 00000100:00100000:3.0:1788574726.486483:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495646464:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574726.486486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.486495:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574726.486500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574726.486515:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495646720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574726.486524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.486527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574726.486529:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.486541:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2942 00000100:00080000:0.0:1788574726.486546:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.486552:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574726.486557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574726.486567:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495646848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574726.486621:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574726.486683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574726.486712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495646848:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788574727.446415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574727.446494:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574728.470334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574728.470351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574729.494382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574729.494753:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574729.558704:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574729.558771:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574729.558794:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574729.558850:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574729.558862:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574729.558874:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574729.558935:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff93a2460d0700 x1875453495646720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574729.558999:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495646720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574729.559015:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574729.559021:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574729.559027:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574729.559036:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff93a2460d0a80 x1875453495646848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574729.559050:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495646848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574729.559058:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574729.559062:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574729.559066:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495646848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574729.559074:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff93a2460d0000 x1875453495646464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574729.559120:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495646464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574729.559128:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574729.559133:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574729.559136:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495646464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574729.559143:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff93a2460d3b80 x1875453495646592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574729.559156:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495646592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574729.559164:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574729.559168:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574729.559171:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495646592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574730.518316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574730.518442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574731.542334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574731.542379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574731.606455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.606467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574731.606471:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.606482:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2948 00000100:00080000:2.0:1788574731.606495:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.606503:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574731.606563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495646976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574731.606590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.606594:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574731.606595:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.606599:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2948 00000100:00080000:2.0:1788574731.606602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.606605:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574731.606616:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574731.606626:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495647104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574731.606634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.606638:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574731.606639:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.606642:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2948 00000100:00080000:2.0:1788574731.606644:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.606650:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574731.606650:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495646976:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574731.606654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574731.606662:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495647232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574731.606670:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.606673:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574731.606675:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.606684:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2948 00000100:00080000:2.0:1788574731.606687:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.606691:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574731.606695:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574731.606705:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495647360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574731.606738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574731.606833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff93a24940ad80 x1875453495646976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574731.606863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495646976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574731.606878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574731.606892:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574731.606897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495646976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574731.606907:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574731.606937:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574731.606953:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff93a249409180 x1875453495647104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574731.606980:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495647104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574731.606999:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574731.607003:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574731.607007:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574731.607041:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647232:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574731.607067:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574731.607081:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff93a249408a80 x1875453495647232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574731.607094:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495647232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574731.607103:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574731.607107:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574731.607110:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647232:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574731.607116:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495647360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574731.607747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574731.607782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff93a249408000 x1875453495647360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574731.607826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495647360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574731.607838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574731.607842:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574731.607846:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495647360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574732.566375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574732.566399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574733.590326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574733.590388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574734.614367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574734.614409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574735.638334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574735.638404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574736.662366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574736.662411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574736.726313:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.726321:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574736.726325:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.726333:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2953 00000100:00080000:2.0:1788574736.726342:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574736.726348:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574736.726380:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495647488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574736.726405:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.726408:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574736.726409:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.726412:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2953 00000100:00080000:2.0:1788574736.726415:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574736.726417:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574736.726425:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574736.726434:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495647616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574736.726442:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.726456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574736.726458:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574736.726460:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495647488:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574736.726464:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2953 00000100:00080000:2.0:1788574736.726470:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574736.726474:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574736.726478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574736.726488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495647744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574736.726495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.726498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574736.726499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.726502:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2953 00000100:00080000:2.0:1788574736.726513:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574736.726516:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574736.726519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574736.726529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495647872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574736.726531:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574736.726578:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff93a24940aa00 x1875453495647488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.726598:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495647488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.726609:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.726613:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574736.726617:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495647488:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574736.726625:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574736.726661:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574736.726675:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff93a249408700 x1875453495647616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.726686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495647616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.726695:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.726698:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574736.726701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574736.726725:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647744:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574736.726748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574736.726760:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff93a249409880 x1875453495647744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.726772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495647744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.726780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.726783:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574736.726787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495647744:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574736.726794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495647872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574736.726815:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574736.726841:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff93a249409500 x1875453495647872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.726866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495647872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.726875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.726878:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574736.726881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495647872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574737.686339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574737.686364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574738.710326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574738.710346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574739.734297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574739.734348:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574740.758355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574740.758370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574741.782283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574741.782378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574741.846397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574741.846409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574741.846414:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574741.846425:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2958 00000100:00080000:0.0:1788574741.846438:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574741.846446:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574741.846494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495648000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574741.846537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574741.846540:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574741.846542:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574741.846545:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2958 00000100:00080000:0.0:1788574741.846548:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574741.846551:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574741.846560:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574741.846571:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495648128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574741.846579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574741.846582:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574741.846583:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574741.846587:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2958 00000100:00080000:0.0:1788574741.846589:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574741.846592:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495648000:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574741.846595:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574741.846604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574741.846610:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495648256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574741.846618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574741.846621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574741.846622:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574741.846625:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2958 00000100:00080000:0.0:1788574741.846633:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574741.846637:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574741.846641:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574741.846650:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495648384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574741.846675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574741.846731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574741] req@ffff93a2460d3b80 x1875453495648000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574741.846749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495648000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574741.846758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574741.846761:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574741.846765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495648000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574741.846790:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574741.846821:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574741.846856:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574741] req@ffff93a2460d0000 x1875453495648128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574741.846873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495648128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574741.846879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574741.846882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574741.846884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574741.846888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648256:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574741.846906:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574741.846914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574741] req@ffff93a2460d0a80 x1875453495648256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574741.846921:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495648256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574741.846926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574741.846928:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574741.846930:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574741.846934:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495648384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574741.846945:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574741.846951:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574741] req@ffff93a2460d0700 x1875453495648384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574741.846958:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495648384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574741.846977:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574741.846981:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574741.846985:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495648384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574742.806316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574742.806334:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574743.830317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574743.830370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574744.854351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574744.854414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574745.878302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574745.878353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574746.902339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574746.902421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574746.966420:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574746.966437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574746.966443:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574746.966457:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2963 00000100:00080000:2.0:1788574746.966471:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574746.966480:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574746.966532:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495648512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574746.966584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574746.966589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574746.966591:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574746.966597:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2963 00000100:00080000:2.0:1788574746.966601:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574746.966607:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574746.966621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574746.966638:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495648640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574746.966649:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574746.966653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574746.966655:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574746.966661:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2963 00000100:00080000:2.0:1788574746.966665:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574746.966674:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574746.966678:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495648512:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574746.966683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574746.966704:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495648768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574746.966719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574746.966724:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574746.966727:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574746.966732:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2963 00000100:00080000:2.0:1788574746.966754:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574746.966760:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788574746.966768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:2.0:1788574746.966773:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574746.966794:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495648896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574746.966839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff93a24940b100 x1875453495648512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.966863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495648512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.966876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.966882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574746.966886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495648512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574746.966896:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574746.966922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574746.966936:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff93a24940bb80 x1875453495648640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.966972:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495648640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.966981:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.966986:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574746.966990:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574746.967016:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648768:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574746.967039:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574746.967051:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff93a24940b480 x1875453495648768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.967064:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495648768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.967074:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.967078:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574746.967082:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495648768:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574746.967089:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495648896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574746.967109:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574746.967121:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff93a249409500 x1875453495648896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.967134:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495648896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.967156:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.967161:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574746.967165:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495648896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574747.926318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574747.926467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574748.950319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574748.950352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574749.974363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574749.974384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574750.998338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574750.998426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574752.022312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574752.022372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574752.086363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574752.086374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574752.086379:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574752.086390:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2968 00000100:00080000:2.0:1788574752.086402:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574752.086410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574752.086449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495649024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574752.086479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574752.086490:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574752.086492:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574752.086497:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2968 00000100:00080000:2.0:1788574752.086502:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574752.086508:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574752.086519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574752.086526:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495649024:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574752.086536:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495649152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574752.086550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574752.086568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574752.086571:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574752.086576:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2968 00000100:00080000:2.0:1788574752.086581:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574752.086582:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574752.086592:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574752.086600:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574752.086615:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495649280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574752.086624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff93a249409880 x1875453495649024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574752.086631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574752.086639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574752.086642:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574752.086646:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495649024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574752.086650:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2968 00000100:00080000:2.0:1788574752.086659:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574752.086662:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574752.086667:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:3.0:1788574752.086670:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574752.086675:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495649024:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574752.086676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574752.086693:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495649408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574752.086699:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574752.086742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574752.086758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff93a249408700 x1875453495649152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574752.086768:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495649152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574752.086782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574752.086785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574752.086787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574752.086792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649280:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574752.086814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574752.086826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff93a24940aa00 x1875453495649280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574752.086840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495649280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574752.086850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574752.086853:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574752.086857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574752.086869:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495649408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574752.086885:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574752.086893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff93a249408000 x1875453495649408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574752.086902:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495649408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574752.086908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574752.086911:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574752.086922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495649408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574753.046299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574753.046365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574754.070331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574754.070386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574755.094307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574755.094355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574756.118298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574756.118363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574757.142285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574757.142357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574757.206343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574757.206352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574757.206357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574757.206367:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2973 00000100:00080000:1.0:1788574757.206377:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574757.206383:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574757.206415:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495649536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574757.206438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574757.206441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574757.206442:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574757.206445:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2973 00000100:00080000:1.0:1788574757.206448:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574757.206450:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574757.206457:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574757.206466:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495649664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574757.206474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574757.206477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574757.206479:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574757.206482:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2973 00000100:00080000:1.0:1788574757.206484:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574757.206486:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574757.206495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495649536:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574757.206505:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574757.206518:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495649792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574757.206527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574757.206530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574757.206532:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574757.206535:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2973 00000100:00080000:1.0:1788574757.206538:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574757.206541:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574757.206544:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574757.206552:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495649920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574757.206572:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574757.206616:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff93a258da2300 x1875453495649536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574757.206637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495649536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574757.206649:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574757.206654:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574757.206659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495649536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574757.206669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574757.206705:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574757.206721:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff93a258da0000 x1875453495649664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574757.206734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495649664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574757.206745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574757.206749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574757.206761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574757.206788:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649792:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574757.206812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574757.206824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff93a258da0380 x1875453495649792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574757.206838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495649792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574757.206847:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574757.206850:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574757.206854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495649792:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574757.206872:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495649920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574757.206895:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574757.206907:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff93a258da1180 x1875453495649920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574757.206936:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495649920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574757.206946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574757.206950:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574757.206954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495649920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574758.166274:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574758.166373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574759.190384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574759.190452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574760.214292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574760.214396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574761.238315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574761.238351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574762.262296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574762.262419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574762.326363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574762.326375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574762.326379:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574762.326390:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2978 00000100:00080000:0.0:1788574762.326400:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574762.326407:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574762.326457:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495650048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574762.326488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574762.326493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574762.326495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574762.326500:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2978 00000100:00080000:0.0:1788574762.326504:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574762.326508:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574762.326517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574762.326528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495650176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574762.326538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574762.326540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574762.326542:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574762.326545:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2978 00000100:00080000:0.0:1788574762.326548:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574762.326550:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574762.326553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574762.326559:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495650304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574762.326568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574762.326571:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574762.326573:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574762.326584:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495650048:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574762.326585:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2978 00000100:00080000:0.0:1788574762.326596:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574762.326601:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574762.326605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574762.326617:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495650432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574762.326674:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574762.326736:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff93a2460d3480 x1875453495650048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574762.326754:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495650048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574762.326776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574762.326780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574762.326784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495650048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574762.326791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574762.326817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574762.326827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff93a2460d1500 x1875453495650176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574762.326838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495650176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574762.326848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574762.326851:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574762.326855:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574762.326890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650304:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574762.326934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574762.326947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff93a2460d2680 x1875453495650304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574762.326959:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495650304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574762.326965:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574762.326968:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574762.326971:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650304:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574762.326975:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495650432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574762.326990:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574762.326996:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff93a2460d0380 x1875453495650432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574762.327004:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495650432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574762.327009:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574762.327010:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574762.327012:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495650432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574763.286393:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574763.286681:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574764.310329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574764.310394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574765.334312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574765.334393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574766.358288:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574766.358344:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574767.382317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574767.382364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574767.446335:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.446347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574767.446352:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.446362:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2983 00000100:00080000:1.0:1788574767.446374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.446383:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574767.446420:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495650560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574767.446449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.446453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574767.446455:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.446459:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2983 00000100:00080000:1.0:1788574767.446480:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.446485:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574767.446496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574767.446515:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495650688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574767.446518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495650560:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574767.446532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.446537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574767.446539:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.446545:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2983 00000100:00080000:1.0:1788574767.446550:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.446555:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574767.446561:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574767.446574:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495650816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574767.446584:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574767.446591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.446596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574767.446599:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.446603:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2983 00000100:00080000:1.0:1788574767.446631:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.446636:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574767.446641:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574767.446656:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495650944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574767.446676:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff93a258da3480 x1875453495650560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574767.446702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495650560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574767.446711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574767.446715:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574767.446718:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495650560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574767.446735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574767.446752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574767.446761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff93a258da2a00 x1875453495650688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574767.446769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495650688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574767.446775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574767.446777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574767.446779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574767.446783:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650816:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574767.446793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574767.446798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff93a258da3100 x1875453495650816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574767.446816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495650816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574767.446822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574767.446824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574767.446826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495650816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574767.446830:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495650944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574767.446842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574767.446848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff93a258da1880 x1875453495650944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574767.446856:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495650944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574767.446861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574767.446863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574767.446864:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495650944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574768.406294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574768.406369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574769.430302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574769.430351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574770.454339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574770.454397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574771.478352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574771.478426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574772.502323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574772.502393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574772.566352:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.566361:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574772.566365:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.566391:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2989 00000100:00080000:3.0:1788574772.566401:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.566410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574772.566446:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495651072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574772.566479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.566482:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574772.566484:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.566489:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2989 00000100:00080000:3.0:1788574772.566494:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.566499:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574772.566510:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574772.566514:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495651072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574772.566530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495651200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574772.566544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.566548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574772.566550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.566554:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2989 00000100:00080000:3.0:1788574772.566559:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.566563:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574772.566570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574772.566574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574772.566587:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495651328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574772.566612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574772.566614:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff93a25064df80 x1875453495651072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574772.566619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574772.566625:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.566633:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2989 00000100:00100000:1.0:1788574772.566638:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495651072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574772.566640:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.566649:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:1.0:1788574772.566651:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574772.566655:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788574772.566657:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574772.566663:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495651072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574772.566671:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495651456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574772.566688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574772.566712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574772.566726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff93a25064c380 x1875453495651200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574772.566741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495651200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.566750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.566753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574772.566757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574772.566764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651328:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574772.566784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574772.566801:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff93a25064ca80 x1875453495651328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574772.566809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495651328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.566815:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.566818:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574772.566820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651328:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574772.566824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495651456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574772.566844:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574772.566856:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff93a25064c700 x1875453495651456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574772.566870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495651456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.566879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.566883:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574772.566886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495651456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574773.526340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574773.526391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574774.550293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574774.550382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574775.574351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574775.574451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574776.598303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574776.598421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574777.622410:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574777.623358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574777.686429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574777.686443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574777.686448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574777.686461:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2994 00000100:00080000:2.0:1788574777.686473:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574777.686483:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574777.686528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495651584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574777.686575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574777.686579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574777.686582:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574777.686587:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2994 00000100:00080000:2.0:1788574777.686591:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574777.686596:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574777.686608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574777.686622:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495651712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574777.686633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574777.686637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574777.686638:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574777.686644:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2994 00000100:00100000:1.0:1788574777.686645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495651584:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574777.686649:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574777.686656:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574777.686662:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574777.686673:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495651840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574777.686684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574777.686688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574777.686690:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574777.686694:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2994 00000100:00080000:2.0:1788574777.686704:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574777.686709:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574777.686715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574777.686727:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495651968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574777.686732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574777.686787:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff93a249408a80 x1875453495651584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.686807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495651584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.686838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.686843:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574777.686847:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495651584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574777.686883:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574777.686909:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574777.686922:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff93a249409180 x1875453495651712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.686933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495651712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.686941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.686945:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574777.686948:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574777.686954:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651840:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574777.686972:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574777.686982:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff93a24940ad80 x1875453495651840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.686993:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495651840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.687000:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.687003:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574777.687006:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495651840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574777.687017:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495651968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574777.687037:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574777.687046:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff93a24940a680 x1875453495651968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.687056:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495651968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.687063:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.687066:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574777.687069:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495651968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574778.646306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574778.646476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574779.670345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574779.670404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574780.694322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574780.694385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574781.718317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574781.718374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574782.742327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574782.742376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574782.806331:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574782.806339:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574782.806343:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574782.806369:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 2999 00000100:00080000:0.0:1788574782.806378:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574782.806384:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574782.806418:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495652096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574782.806444:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574782.806448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574782.806450:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574782.806454:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2999 00000100:00080000:0.0:1788574782.806458:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574782.806462:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574782.806471:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574782.806481:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495652224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574782.806489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574782.806491:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495652096:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574782.806492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574782.806495:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574782.806501:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 2999 00000100:00080000:0.0:1788574782.806503:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574782.806505:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574782.806508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574782.806517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495652352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574782.806532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574782.806535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574782.806536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574782.806540:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 2999 00000100:00080000:0.0:1788574782.806545:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574782.806549:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788574782.806552:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788574782.806558:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574782.806573:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495652480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574782.806588:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff93a2460d2300 x1875453495652096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.806603:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495652096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.806610:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.806613:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574782.806615:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495652096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574782.806633:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574782.806646:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574782.806653:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff93a2460d1f80 x1875453495652224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.806667:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495652224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.806673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.806675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574782.806677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574782.806683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652352:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574782.806701:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574782.806719:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff93a2460d2a00 x1875453495652352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.806733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495652352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.806739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.806741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574782.806743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652352:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574782.806747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495652480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574782.806761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574782.806767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff93a2460d1180 x1875453495652480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.806774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495652480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.806788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.806790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574782.806792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495652480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574783.766351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574783.766408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574784.790297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574784.790352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574785.814325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574785.814378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574786.838279:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574786.838354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574787.862316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574787.862371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574787.926297:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574787.926305:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574787.926309:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574787.926318:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3004 00000100:00080000:0.0:1788574787.926327:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574787.926335:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574787.926367:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495652608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574787.926387:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574787.926390:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574787.926402:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574787.926406:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3004 00000100:00080000:0.0:1788574787.926409:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574787.926414:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574787.926416:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495652608:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574787.926423:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574787.926435:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495652736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574787.926442:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574787.926445:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574787.926446:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574787.926450:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3004 00000100:00080000:0.0:1788574787.926461:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574787.926463:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574787.926466:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574787.926472:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495652864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574787.926478:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574787.926486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574787.926489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574787.926490:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574787.926493:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3004 00000100:00080000:0.0:1788574787.926496:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574787.926498:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574787.926500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574787.926514:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495652992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574787.926520:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff93a2460d1180 x1875453495652608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.926537:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495652608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.926544:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.926547:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574787.926550:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495652608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574787.926568:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574787.926585:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574787.926601:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff93a2460d2a00 x1875453495652736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.926608:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495652736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.926614:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.926616:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574787.926618:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574787.926628:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652864:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574787.926650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574787.926661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff93a2460d1f80 x1875453495652864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.926672:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495652864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.926680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.926683:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574787.926686:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495652864:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574787.926691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495652992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574787.926719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574787.926730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff93a2460d2300 x1875453495652992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.926749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495652992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.926756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.926759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574787.926762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495652992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574788.886303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574788.886354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574789.910330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574789.910383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574790.934279:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574790.934327:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574791.958305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574791.958360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574792.982444:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574792.982614:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574793.046372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574793.046381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574793.046385:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574793.046410:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3009 00000100:00080000:1.0:1788574793.046418:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574793.046424:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574793.046452:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495653120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574793.046479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574793.046483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574793.046485:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574793.046489:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3009 00000100:00080000:1.0:1788574793.046493:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574793.046497:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574793.046507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574793.046520:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495653248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574793.046531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574793.046535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574793.046536:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574793.046541:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3009 00000100:00080000:1.0:1788574793.046544:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574793.046544:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495653120:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574793.046549:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574793.046554:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574793.046565:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495653376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574793.046586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574793.046590:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574793.046592:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574793.046596:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3009 00000100:00080000:1.0:1788574793.046599:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574793.046603:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574793.046608:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574793.046619:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495653504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574793.046635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574793.046690:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653376:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574793.046723:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495653504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574794.006332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574794.006767:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574795.030341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574795.030529:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574796.054582:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574796.054614:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:0.0:1788574796.118584:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788574796.118653:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574796.118677:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574796.118720:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574796.118727:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574796.118773:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574796.118796:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff93a258da3800 x1875453495653376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.118829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495653376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.118849:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.118854:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574796.118858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653376:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574796.118867:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff93a258da1c00 x1875453495653504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.118882:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495653504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.118891:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.118926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574796.118931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495653504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574796.118940:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff93a258da0700 x1875453495653120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.118957:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495653120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.118966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.118970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574796.118973:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495653120:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574796.118982:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff93a258da0a80 x1875453495653248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.118995:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495653248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.119006:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.119009:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574796.119012:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574797.078582:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574797.078641:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574798.102317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574798.102367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574798.166410:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574798.166421:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574798.166425:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574798.166433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3014 00000100:00080000:0.0:1788574798.166441:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574798.166449:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574798.166485:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495653632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574798.166522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574798.166527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574798.166529:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574798.166534:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3014 00000100:00080000:0.0:1788574798.166539:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574798.166561:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574798.166571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574798.166585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495653760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574798.166596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574798.166599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574798.166601:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574798.166606:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3014 00000100:00080000:0.0:1788574798.166610:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574798.166614:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574798.166619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574798.166630:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495653888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574798.166639:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574798.166642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574798.166644:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574798.166647:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3014 00000100:00080000:0.0:1788574798.166650:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574798.166654:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574798.166658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574798.166682:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495654016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574798.166809:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495653632:192.168.202.102@tcp:250:kworker.0 00000800:00000100:3.0:1788574798.166876:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574798.166930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff93a2460d0380 x1875453495653632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.166964:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495653632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.166974:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.166979:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574798.166983:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495653632:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574798.166993:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574798.167019:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574798.167034:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff93a2460d2680 x1875453495653760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.167046:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495653760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.167055:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.167059:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574798.167062:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574798.167068:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653888:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574798.167091:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574798.167104:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff93a2460d1500 x1875453495653888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.167117:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495653888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.167302:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.167307:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574798.167311:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495653888:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574798.167319:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495654016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574798.167345:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574798.167359:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff93a2460d3480 x1875453495654016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.167372:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495654016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.167381:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.167384:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574798.167388:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495654016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574799.126334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574799.126395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574800.150379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574800.150477:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574801.174387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574801.174465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574802.198347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574802.198410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574803.222297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574803.222622:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574803.286428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.286451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574803.286456:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.286482:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3019 00000100:00080000:1.0:1788574803.286504:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.286530:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574803.286591:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495654144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574803.286645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.286648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574803.286650:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.286653:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3019 00000100:00080000:1.0:1788574803.286665:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.286669:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574803.286702:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495654144:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574803.286707:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574803.286746:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495654272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574803.286759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.286762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574803.286764:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.286768:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3019 00000100:00080000:1.0:1788574803.286785:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.286790:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574803.286796:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574803.286816:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574803.286825:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495654400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574803.286877:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff93a258da2680 x1875453495654144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574803.286878:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.286893:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574803.286904:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.286916:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3019 00000100:00100000:3.0:1788574803.286917:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495654144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574803.286923:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.286927:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574803.286943:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788574803.286943:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.286956:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574803.286963:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495654528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574803.286968:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495654144:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574803.287017:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574803.287047:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574803.287059:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff93a258da0e00 x1875453495654272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.287069:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495654272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.287075:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.287078:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574803.287080:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574803.287084:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654400:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574803.287099:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574803.287106:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff93a258da1f80 x1875453495654400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.287113:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495654400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.287118:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.287120:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574803.287122:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654400:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574803.287133:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495654528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574803.287154:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574803.287166:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff93a258da0a80 x1875453495654528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.287179:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495654528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.287188:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.287192:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574803.287196:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495654528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574804.246322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574804.246394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574805.270304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574805.270377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574806.294301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574806.294402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574807.318321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574807.318386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574808.342285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574808.342372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574808.406382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.406409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574808.406414:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.406433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3024 00000100:00080000:0.0:1788574808.406445:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.406453:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574808.406488:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495654656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.406518:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.406522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.406524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.406529:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3024 00000100:00080000:0.0:1788574808.406533:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.406537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574808.406555:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574808.406569:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495654784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574808.406579:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495654656:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574808.406585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.406590:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.406592:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.406598:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3024 00000100:00080000:0.0:1788574808.406603:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.406607:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574808.406613:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574808.406627:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495654912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.406646:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574808.406648:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574808.406654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.406657:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.406664:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3024 00000100:00080000:0.0:1788574808.406681:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.406686:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574808.406693:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574808.406707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff93a2460d0700 x1875453495654656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574808.406712:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495655040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574808.406737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495654656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.406746:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.406749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574808.406753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495654656:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574808.406782:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574808.406801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574808.406811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff93a2460d0a80 x1875453495654784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.406820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495654784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.406826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.406829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574808.406831:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574808.406835:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654912:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574808.406846:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574808.406855:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff93a2460d0000 x1875453495654912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.406867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495654912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.406876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.406880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574808.406883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495654912:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574808.406899:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495655040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574808.406920:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574808.406933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff93a2460d3b80 x1875453495655040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.406953:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495655040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.406961:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.406964:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574808.406966:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495655040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574809.366277:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574809.366334:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574810.390306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574810.390340:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574811.414323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574811.414423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574812.438348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574812.438378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574813.462335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574813.462437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574813.526372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.526383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574813.526387:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.526398:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3029 00000100:00080000:0.0:1788574813.526409:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.526416:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574813.526460:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495655168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.526489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.526493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574813.526494:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.526498:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3029 00000100:00100000:3.0:1788574813.526515:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495655168:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574813.526519:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.526526:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574813.526537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574813.526553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495655296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.526563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.526566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574813.526569:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574813.526570:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.526578:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3029 00000100:00080000:0.0:1788574813.526583:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.526588:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574813.526593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574813.526605:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff93a2460d3b80 x1875453495655168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574813.526608:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495655424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.526625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574813.526626:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495655168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574813.526635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574813.526639:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574813.526639:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.526646:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:0.0:1788574813.526647:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3029 00000100:00100000:3.0:1788574813.526650:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495655168:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574813.526653:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.526660:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574813.526665:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574813.526676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574813.526679:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495655552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574813.526705:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574813.526727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff93a2460d0000 x1875453495655296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.526747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495655296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.526753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.526755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574813.526757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574813.526761:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655424:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574813.526777:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574813.526788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff93a2460d0a80 x1875453495655424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.526799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495655424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.526806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.526809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574813.526813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655424:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574813.526828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495655552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574813.526849:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574813.526860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff93a2460d0700 x1875453495655552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.526872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495655552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.526891:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.526895:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574813.526899:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495655552:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788574814.486363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574814.486363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574815.510295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574815.510364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574816.534309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574816.534354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574817.558325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574817.558374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574818.582323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574818.582392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574818.646261:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.646271:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574818.646275:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.646284:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3035 00000100:00080000:0.0:1788574818.646295:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.646302:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574818.646332:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495655680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574818.646356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.646358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574818.646360:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.646363:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3035 00000100:00080000:0.0:1788574818.646365:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.646367:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574818.646374:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574818.646382:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495655808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574818.646389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.646392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788574818.646400:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495655680:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574818.646406:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.646411:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3035 00000100:00080000:0.0:1788574818.646413:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.646415:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574818.646427:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574818.646440:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495655936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574818.646452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574818.646452:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574818.646459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574818.646462:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.646467:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3035 00000100:00080000:0.0:1788574818.646473:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.646477:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574818.646483:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574818.646495:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff93a2460d3480 x1875453495655680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574818.646497:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495656064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574818.646520:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495655680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.646528:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.646532:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574818.646535:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495655680:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574818.646540:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574818.646553:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574818.646560:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff93a2460d1500 x1875453495655808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.646567:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495655808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.646577:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.646579:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574818.646582:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574818.646598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655936:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574818.646611:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574818.646617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff93a2460d2680 x1875453495655936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.646623:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495655936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.646628:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.646630:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574818.646632:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495655936:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574818.646634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495656064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574818.646642:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574818.646649:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff93a2460d0380 x1875453495656064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.646659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495656064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.646666:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.646669:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574818.646677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495656064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574819.606273:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574819.606361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574820.630330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574820.630397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574821.654306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574821.654388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574822.678320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574822.678403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574823.702296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574823.702352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574823.766319:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.766327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574823.766346:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.766356:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3040 00000100:00080000:1.0:1788574823.766364:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.766370:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574823.766399:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495656192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.766421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.766423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574823.766425:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.766428:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3040 00000100:00080000:1.0:1788574823.766430:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.766433:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574823.766441:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574823.766448:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495656320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.766455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.766457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574823.766458:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.766461:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3040 00000100:00080000:1.0:1788574823.766463:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.766465:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574823.766468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574823.766474:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495656448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.766491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.766494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574823.766496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574823.766499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495656192:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574823.766502:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3040 00000100:00080000:1.0:1788574823.766509:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.766512:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574823.766516:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574823.766529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495656576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574823.766562:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574823.766612:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff93a258da0700 x1875453495656192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.766634:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495656192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.766646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.766651:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574823.766655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495656192:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574823.766664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574823.766690:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574823.766704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff93a258da1c00 x1875453495656320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.766717:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495656320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.766724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.766727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574823.766736:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574823.766756:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656448:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574823.766774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574823.766782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff93a258da3800 x1875453495656448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.766790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495656448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.766795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.766798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574823.766800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656448:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574823.766803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495656576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574823.766812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574823.766818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff93a258da1880 x1875453495656576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.766825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495656576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.766830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.766832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574823.766834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495656576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574824.726326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574824.726411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574825.750311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574825.750373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574826.774354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574826.774440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574827.798340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574827.798427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574828.822374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574828.822435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574828.886322:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574828.886332:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574828.886335:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574828.886344:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3045 00000100:00080000:1.0:1788574828.886353:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574828.886359:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574828.886391:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495656704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574828.886415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574828.886417:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574828.886419:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574828.886422:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3045 00000100:00080000:1.0:1788574828.886424:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574828.886426:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574828.886447:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574828.886458:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495656832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574828.886467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574828.886469:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574828.886471:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574828.886475:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3045 00000100:00080000:1.0:1788574828.886478:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574828.886480:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574828.886484:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574828.886493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495656960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574828.886504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574828.886506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495656704:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574828.886508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574828.886514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574828.886527:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3045 00000100:00080000:1.0:1788574828.886530:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574828.886533:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574828.886539:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574828.886553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495657088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574828.886614:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574828.886671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff93a258da3100 x1875453495656704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574828.886689:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495656704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574828.886698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574828.886702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574828.886704:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495656704:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574828.886712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574828.886735:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574828.886758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff93a258da2a00 x1875453495656832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574828.886769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495656832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574828.886775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574828.886777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574828.886779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574828.886806:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656960:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574828.886866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574828.886882:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff93a258da3480 x1875453495656960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574828.886890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495656960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574828.886896:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574828.886898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574828.886900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495656960:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574828.886904:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495657088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574828.886917:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574828.886924:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff93a258da1180 x1875453495657088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574828.886931:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495657088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574828.886935:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574828.886937:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574828.886938:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495657088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574829.846375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574829.846476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574830.870296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574830.870372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574831.894332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574831.894423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574832.918301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574832.918359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574833.942329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574833.942412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574834.006366:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.006375:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574834.006379:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.006387:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3050 00000100:00080000:0.0:1788574834.006396:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.006402:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574834.006434:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495657216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.006465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.006469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574834.006471:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.006476:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3050 00000100:00080000:0.0:1788574834.006480:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.006484:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574834.006496:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574834.006510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495657344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.006519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.006522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574834.006523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.006541:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3050 00000100:00100000:3.0:1788574834.006553:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495657216:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574834.006554:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.006562:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574834.006568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574834.006579:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495657472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.006595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.006599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574834.006601:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.006606:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3050 00000100:00080000:0.0:1788574834.006611:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.006616:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574834.006623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574834.006638:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495657600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574834.006661:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574834.006730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff93a2460d2300 x1875453495657216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.006751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495657216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.006762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.006767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574834.006771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495657216:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574834.006777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574834.006796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574834.006806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff93a2460d1f80 x1875453495657344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.006813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495657344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.006818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.006820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574834.006822:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574834.006845:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657472:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574834.006858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574834.006864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff93a2460d2a00 x1875453495657472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.006889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495657472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.006897:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.006900:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574834.006911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657472:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574834.006919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495657600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574834.006941:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574834.006950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff93a2460d1180 x1875453495657600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.006958:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495657600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.006963:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.006966:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574834.006968:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495657600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574834.966314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574834.966368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574835.990283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574835.990379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574837.014327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574837.014383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574838.038305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574838.038371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574839.062370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574839.062431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574839.126322:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574839.126331:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574839.126335:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574839.126343:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3055 00000100:00080000:0.0:1788574839.126359:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574839.126365:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574839.126415:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495657728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574839.126438:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574839.126441:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574839.126443:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574839.126446:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3055 00000100:00080000:0.0:1788574839.126449:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574839.126452:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574839.126460:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574839.126468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495657856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574839.126475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574839.126478:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574839.126480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574839.126483:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3055 00000100:00080000:0.0:1788574839.126485:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574839.126487:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574839.126492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574839.126499:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495657984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574839.126506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574839.126509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574839.126510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574839.126525:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3055 00000100:00080000:0.0:1788574839.126528:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574839.126537:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574839.126537:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495657728:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574839.126552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574839.126585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495658112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574839.126644:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574839.126698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff93a2460d1180 x1875453495657728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.126742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495657728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.126752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.126756:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574839.126760:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495657728:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574839.126766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574839.126785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574839.126795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff93a2460d2a00 x1875453495657856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.126803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495657856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.126808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.126810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574839.126812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574839.126816:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657984:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574839.126827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574839.126833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff93a2460d1f80 x1875453495657984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.126839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495657984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.126853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.126855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574839.126857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495657984:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574839.126883:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495658112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574839.126896:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574839.126903:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff93a2460d2300 x1875453495658112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.126910:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495658112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.126914:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.126917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574839.126918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495658112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574840.086302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574840.086402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574841.110387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574841.110457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574842.134374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574842.134440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574843.158326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574843.158391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574844.182282:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574844.182366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574844.246353:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.246364:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574844.246369:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.246380:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3060 00000100:00080000:0.0:1788574844.246391:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.246399:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574844.246453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495658240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.246487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.246491:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574844.246493:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.246498:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3060 00000100:00080000:0.0:1788574844.246502:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.246507:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574844.246526:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574844.246540:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495658240:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574844.246545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495658368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.246560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.246579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574844.246582:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.246588:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3060 00000100:00080000:0.0:1788574844.246594:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.246600:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574844.246607:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574844.246613:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574844.246626:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495658496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.246640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.246643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574844.246646:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.246651:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3060 00000100:00080000:0.0:1788574844.246655:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.246659:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574844.246667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff93a2460d0380 x1875453495658240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574844.246672:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574844.246688:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495658240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574844.246692:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495658624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574844.246726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.246731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574844.246735:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495658240:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574844.246755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495658368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574844.246774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574844.246783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff93a2460d2680 x1875453495658368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.246791:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495658368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.246797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.246799:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574844.246802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495658368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574844.246805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495658496:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574844.246817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574844.246823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff93a2460d1500 x1875453495658496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.246830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495658496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.246835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.246837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574844.246839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495658496:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574844.246848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495658624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574844.246861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574844.246868:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff93a2460d3480 x1875453495658624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.246876:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495658624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.246881:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.246883:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574844.246885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495658624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574845.206322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574845.206368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574846.230346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574846.230449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574847.254316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574847.254356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574848.278371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574848.278458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574849.302365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574849.302435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574849.366335:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.366347:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574849.366353:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.366365:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3065 00000100:00080000:0.0:1788574849.366378:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.366386:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574849.366427:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495658752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.366472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.366475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574849.366477:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.366482:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3065 00000100:00080000:0.0:1788574849.366485:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.366490:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574849.366510:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574849.366510:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495658752:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574849.366527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495658880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.366538:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.366542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574849.366544:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.366548:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3065 00000100:00080000:0.0:1788574849.366552:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.366555:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574849.366561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574849.366572:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495659008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.366581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574849.366587:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574849.366588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574849.366591:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.366596:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3065 00000100:00080000:0.0:1788574849.366609:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.366613:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574849.366618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574849.366630:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495659136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574849.366635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff93a2460d0700 x1875453495658752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.366661:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495658752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.366670:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.366674:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574849.366677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495658752:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574849.366705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495658880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574849.366721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574849.366730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff93a2460d0a80 x1875453495658880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.366737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495658880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.366742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.366744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574849.366746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495658880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574849.366749:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659008:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574849.366760:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574849.366765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff93a2460d0000 x1875453495659008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.366771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495659008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.366775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.366777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574849.366778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659008:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574849.366781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495659136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574849.366790:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574849.366795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff93a2460d3b80 x1875453495659136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.366806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495659136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.366811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.366812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574849.366814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495659136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574850.326325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574850.326413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574851.350377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574851.350459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574852.374294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574852.374394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574853.398363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574853.398420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574854.422359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574854.422435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574854.486326:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574854.486342:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574854.486348:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574854.486363:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3070 00000100:00080000:0.0:1788574854.486378:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574854.486388:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574854.486437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495659264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574854.486473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574854.486477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574854.486480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574854.486484:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3070 00000100:00080000:0.0:1788574854.486489:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574854.486494:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574854.486507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574854.486523:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495659392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574854.486537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574854.486542:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495659264:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574854.486571:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574854.486573:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574854.486580:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3070 00000100:00080000:0.0:1788574854.486584:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574854.486589:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574854.486595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574854.486610:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495659520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574854.486623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574854.486627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574854.486629:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574854.486634:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3070 00000100:00080000:0.0:1788574854.486646:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574854.486652:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574854.486658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574854.486658:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574854.486675:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495659648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574854.486733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff93a2460d3b80 x1875453495659264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.486759:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495659264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.486773:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.486778:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574854.486783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495659264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574854.486814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574854.486842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574854.486858:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff93a2460d0000 x1875453495659392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.486882:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495659392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.486892:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.486896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574854.486900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574854.486907:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659520:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574854.486934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574854.486948:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff93a2460d0a80 x1875453495659520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.486962:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495659520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.486970:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.486974:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574854.486978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659520:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574854.486985:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495659648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574854.487007:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574854.487020:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff93a2460d0700 x1875453495659648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.487034:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495659648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.487056:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.487060:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574854.487064:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495659648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574855.446333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574855.446401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574856.470328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574856.470387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574857.494381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574857.494470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574858.518311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574858.518364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574859.542339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574859.542742:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574859.606341:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574859.606349:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574859.606353:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574859.606361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3076 00000100:00080000:2.0:1788574859.606370:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574859.606376:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574859.606406:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495659776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574859.606430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574859.606433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574859.606434:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574859.606437:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3076 00000100:00080000:2.0:1788574859.606439:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574859.606441:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574859.606449:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574859.606473:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495659904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574859.606480:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574859.606483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574859.606484:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574859.606487:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3076 00000100:00080000:2.0:1788574859.606490:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574859.606492:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574859.606497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495659776:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574859.606498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574859.606510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495660032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574859.606518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574859.606520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574859.606521:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574859.606524:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3076 00000100:00080000:2.0:1788574859.606526:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574859.606529:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574859.606540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574859.606553:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495660160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574859.606595:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574859.606624:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660032:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574859.606653:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495660160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574860.566331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574860.566762:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574861.590772:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574861.590895:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574862.614662:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574862.615607:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788574862.678506:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788574862.678550:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574862.678568:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574862.678693:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff93a249408e00 x1875453495659776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574884 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.678726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495659776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574884 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.678740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.678745:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574862.678750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495659776:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788574862.678820:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574862.678845:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788574862.678856:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574862.679065:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff93a249409c00 x1875453495660032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.679092:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495660032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.679102:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.679107:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574862.679111:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660032:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574862.679119:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff93a24940a680 x1875453495660160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.679134:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495660160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.679142:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.679146:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574862.679150:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495660160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574862.679157:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff93a24940b800 x1875453495659904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.679170:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495659904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.679179:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.679182:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574862.679186:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495659904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574863.638644:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574863.638765:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574864.662348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574864.662357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574864.726354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574864.726363:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574864.726366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574864.726374:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3081 00000100:00080000:0.0:1788574864.726384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574864.726391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574864.726419:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495660288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574864.726443:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574864.726447:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574864.726450:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574864.726455:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3081 00000100:00080000:0.0:1788574864.726460:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574864.726466:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574864.726477:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574864.726488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495660288:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574864.726495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495660416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574864.726524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574864.726529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574864.726531:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574864.726536:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3081 00000100:00080000:0.0:1788574864.726550:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574864.726557:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788574864.726559:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788574864.726568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574864.726599:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495660544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574864.726610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff93a2460d3480 x1875453495660288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574864.726616:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574864.726626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574864.726630:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574864.726638:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3081 00000100:00080000:0.0:1788574864.726643:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574864.726646:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495660288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574864.726652:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574864.726663:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788574864.726668:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.726675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788574864.726682:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495660672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574864.726683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495660288:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574864.726717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574864.726749:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574864.726763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff93a2460d1500 x1875453495660416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.726778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495660416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.726787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.726791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574864.726794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574864.726802:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660544:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574864.726824:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574864.726836:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff93a2460d2680 x1875453495660544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.726866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495660544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.726876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.726879:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574864.726883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660544:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574864.726890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495660672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574864.726915:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574864.726943:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff93a2460d0380 x1875453495660672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.726955:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495660672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.726963:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.726967:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574864.726970:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495660672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574865.686370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574865.686452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574866.710351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574866.710456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574867.734351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574867.734439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574868.758317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574868.758404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574869.782343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574869.782411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574869.846317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574869.846329:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574869.846334:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574869.846383:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3086 00000100:00080000:3.0:1788574869.846399:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574869.846409:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574869.846450:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495660800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574869.846479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574869.846485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574869.846487:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574869.846490:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3086 00000100:00080000:3.0:1788574869.846493:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574869.846496:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574869.846507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574869.846517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495660928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574869.846525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574869.846528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574869.846530:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574869.846534:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3086 00000100:00080000:3.0:1788574869.846536:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574869.846538:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574869.846542:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574869.846550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495661056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574869.846572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495660800:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574869.846573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574869.846577:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574869.846579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574869.846583:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3086 00000100:00080000:3.0:1788574869.846586:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574869.846590:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574869.846602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574869.846623:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495661184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574869.846660:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574869.846698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff93a25064d500 x1875453495660800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.846728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495660800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.846737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.846741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574869.846745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495660800:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574869.846751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574869.846771:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574869.846784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff93a25064e300 x1875453495660928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.846811:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495660928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.846821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.846825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574869.846829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495660928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574869.846856:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661056:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574869.846874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574869.846883:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff93a25064ce00 x1875453495661056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.846902:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495661056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.846908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.846910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574869.846912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661056:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574869.846917:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495661184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574869.846931:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574869.846938:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff93a25064f100 x1875453495661184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.846946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495661184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.846951:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.846953:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574869.846955:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495661184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574870.806352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574870.806424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574871.830334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574871.830425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574872.854329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574872.854432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574873.878295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574873.878380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574874.902358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574874.902420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574874.966375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574874.966391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574874.966397:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574874.966409:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3091 00000100:00080000:1.0:1788574874.966421:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574874.966428:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574874.966469:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495661312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574874.966511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574874.966513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574874.966515:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574874.966518:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3091 00000100:00080000:1.0:1788574874.966521:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574874.966524:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574874.966534:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574874.966545:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495661312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574874.966546:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495661440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574874.966558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574874.966561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574874.966561:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574874.966564:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3091 00000100:00080000:1.0:1788574874.966566:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574874.966568:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574874.966571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574874.966579:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495661568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574874.966586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574874.966588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574874.966589:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574874.966591:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3091 00000100:00080000:1.0:1788574874.966598:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574874.966601:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574874.966605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574874.966615:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495661696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574874.966623:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574874.966666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574874] req@ffff93a258da0380 x1875453495661312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574874.966702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495661312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574874.966711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574874.966715:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574874.966719:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495661312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574874.966754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574874.966775:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574874.966784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574874] req@ffff93a258da0000 x1875453495661440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574874.966792:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495661440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574874.966798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574874.966800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574874.966802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574874.966806:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661568:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574874.966817:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574874.966823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574874] req@ffff93a258da2300 x1875453495661568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574874.966829:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495661568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574874.966835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574874.966838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574874.966841:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574874.966846:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495661696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574874.966865:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574874.966875:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574874] req@ffff93a258da1180 x1875453495661696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574874.966892:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495661696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574874.966898:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574874.966900:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574874.966902:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495661696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574875.926333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574875.926408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574876.950323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574876.950398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574877.974335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574877.974425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574878.998317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574878.998371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574880.022325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574880.022395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574880.086343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574880.086353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574880.086357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574880.086367:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3096 00000100:00080000:1.0:1788574880.086377:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574880.086382:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574880.086415:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495661824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574880.086436:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574880.086439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574880.086440:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574880.086443:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3096 00000100:00080000:1.0:1788574880.086445:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574880.086448:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574880.086456:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574880.086467:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495661952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574880.086475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574880.086482:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495661824:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574880.086491:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574880.086494:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574880.086498:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3096 00000100:00080000:1.0:1788574880.086503:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574880.086507:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574880.086512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574880.086525:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495662080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574880.086535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574880.086539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574880.086542:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574880.086543:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574880.086550:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3096 00000100:00080000:1.0:1788574880.086555:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574880.086559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574880.086564:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574880.086588:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495662208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574880.086602:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff93a258da3480 x1875453495661824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.086621:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495661824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.086632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.086636:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574880.086640:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495661824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574880.086661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574880.086683:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574880.086696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff93a258da2a00 x1875453495661952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.086715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495661952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.086723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.086727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574880.086730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495661952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574880.086735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662080:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574880.086755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574880.086765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff93a258da3100 x1875453495662080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.086778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495662080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.086786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.086790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574880.086793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662080:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574880.086799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495662208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574880.086816:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574880.086842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff93a258da1880 x1875453495662208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.086851:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495662208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.086871:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.086873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574880.086875:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495662208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574881.046296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574881.046341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574882.070326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574882.070382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574883.094318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574883.094368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574884.118359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574884.118436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574885.142273:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574885.142357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574885.206379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574885.206388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574885.206392:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574885.206400:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3101 00000100:00080000:2.0:1788574885.206408:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574885.206414:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574885.206448:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495662336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574885.206489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574885.206510:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574885.206512:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574885.206516:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3101 00000100:00080000:2.0:1788574885.206521:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574885.206525:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574885.206548:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495662336:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574885.206549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574885.206571:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495662464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574885.206586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574885.206590:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574885.206592:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574885.206597:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3101 00000100:00080000:2.0:1788574885.206608:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574885.206612:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788574885.206614:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574885.206621:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574885.206632:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495662592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574885.206640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574885.206643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574885.206645:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574885.206648:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3101 00000100:00080000:2.0:1788574885.206651:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574885.206653:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574885.206664:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574885.206665:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff93a24940ad80 x1875453495662336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574885.206674:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495662720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574885.206683:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495662336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.206691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.206695:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574885.206698:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495662336:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574885.206717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574885.206735:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574885.206744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff93a249409180 x1875453495662464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.206751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495662464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.206764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.206766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574885.206768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574885.206772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662592:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574885.206786:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574885.206793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff93a249408a80 x1875453495662592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.206800:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495662592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.206805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.206807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574885.206809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574885.206812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495662720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574885.206822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574885.206827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff93a249408000 x1875453495662720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.206834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495662720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.206839:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.206841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574885.206847:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495662720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574886.166352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574886.166393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574887.190362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574887.190430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574888.214374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574888.214455:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574889.238309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574889.238423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574890.262369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574890.262458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574890.326408:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574890.326424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574890.326430:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574890.326444:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3106 00000100:00080000:2.0:1788574890.326458:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574890.326467:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574890.326553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495662848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574890.326588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574890.326592:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574890.326594:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574890.326599:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3106 00000100:00080000:2.0:1788574890.326603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574890.326608:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574890.326621:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574890.326637:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495662976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574890.326639:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495662848:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574890.326680:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574890.326684:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574890.326686:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574890.326692:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3106 00000100:00080000:2.0:1788574890.326706:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574890.326711:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574890.326717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574890.326726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574890.326733:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495663104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574890.326745:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574890.326749:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574890.326752:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574890.326756:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3106 00000100:00080000:2.0:1788574890.326760:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574890.326765:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574890.326771:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574890.326785:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495663232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574890.326792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff93a24940aa00 x1875453495662848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574890.326815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495662848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574890.326825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574890.326828:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574890.326832:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495662848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574890.326858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574890.326877:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574890.326885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff93a249408700 x1875453495662976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574890.326893:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495662976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574890.326899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574890.326901:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574890.326911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495662976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574890.326915:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495663104:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574890.326931:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574890.326938:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff93a249409880 x1875453495663104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574890.326946:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495663104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574890.326951:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574890.326952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574890.326954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495663104:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574890.326958:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495663232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574890.326969:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574890.326975:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff93a249409500 x1875453495663232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574890.326982:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495663232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574890.326987:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574890.326988:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574890.326990:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495663232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574891.286314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574891.286381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574892.310375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574892.310457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574893.334352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574893.334458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574894.358386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574894.358615:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574895.382325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574895.382405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574895.446333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.446344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574895.446349:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.446359:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3111 00000100:00080000:0.0:1788574895.446371:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.446378:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574895.446424:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495663360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574895.446449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.446452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574895.446454:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.446472:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3111 00000100:00080000:0.0:1788574895.446475:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.446498:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574895.446510:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574895.446528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495663488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574895.446529:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495663360:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574895.446552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.446555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574895.446557:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.446563:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3111 00000100:00080000:0.0:1788574895.446569:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.446571:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574895.446575:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574895.446591:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495663616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574895.446599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.446602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574895.446604:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.446608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3111 00000100:00080000:0.0:1788574895.446610:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.446613:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574895.446619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574895.446624:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574895.446633:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495663744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574895.446676:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff93a2460d2300 x1875453495663360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.446693:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495663360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.446702:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.446706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574895.446710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495663360:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574895.446731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495663488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574895.446748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574895.446756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff93a2460d1f80 x1875453495663488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.446769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495663488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.446775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.446777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574895.446779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495663488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574895.446783:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495663616:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574895.446814:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574895.446822:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff93a2460d2a00 x1875453495663616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.446830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495663616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.446835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.446837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574895.446839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495663616:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574895.446843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495663744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574895.446854:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574895.446860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff93a2460d1180 x1875453495663744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.446867:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495663744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.446872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.446874:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574895.446876:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495663744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574896.406339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574896.406422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574897.430295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574897.430356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574898.454322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574898.454387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574899.478312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574899.478375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574900.502314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574900.502370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574900.566321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574900.566349:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574900.566354:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574900.566373:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3117 00000100:00080000:3.0:1788574900.566384:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574900.566392:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574900.566440:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495663872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574900.566468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574900.566471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574900.566473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574900.566476:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3117 00000100:00080000:3.0:1788574900.566481:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574900.566486:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574900.566497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574900.566513:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495664000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574900.566522:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495663872:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788574900.566526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574900.566533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574900.566535:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574900.566541:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3117 00000100:00080000:3.0:1788574900.566545:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574900.566550:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574900.566555:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574900.566570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495664128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574900.566590:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788574900.566595:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788574900.566597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574900.566601:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574900.566606:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3117 00000100:00080000:3.0:1788574900.566612:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574900.566617:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788574900.566622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574900.566638:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495664256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574900.566644:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff93a25064d180 x1875453495663872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.566668:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495663872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.566693:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.566697:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574900.566701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495663872:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574900.566728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574900.566753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574900.566766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff93a25064c000 x1875453495664000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.566788:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495664000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.566797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.566801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574900.566804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574900.566811:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664128:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574900.566835:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574900.566848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff93a25064f480 x1875453495664128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.566861:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495664128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.566869:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.566873:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574900.566877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574900.566890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495664256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574900.566924:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574900.566938:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff93a25064fb80 x1875453495664256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.566951:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495664256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.566961:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.566964:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574900.566967:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495664256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574901.526316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574901.526366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574902.550317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574902.550378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574903.574368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574903.574439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574904.598350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574904.598427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574905.622402:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574905.622484:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574905.686456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.686497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574905.686502:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.686516:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3122 00000100:00080000:0.0:1788574905.686532:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.686551:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574905.686595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495664384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.686623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.686626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.686628:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.686632:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3122 00000100:00080000:0.0:1788574905.686635:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.686638:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574905.686648:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.686659:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495664512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.686667:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.686670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.686671:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.686677:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3122 00000100:00080000:0.0:1788574905.686679:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574905.686679:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495664384:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574905.686686:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574905.686691:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.686700:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495664640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.686720:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.686724:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.686726:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.686731:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3122 00000100:00080000:0.0:1788574905.686747:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574905.686754:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574905.686762:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574905.686769:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.686781:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495664768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574905.686805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff93a2460d1180 x1875453495664384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.686821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495664384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.686830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.686833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574905.686836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495664384:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574905.686859:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574905.686875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574905.686884:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff93a2460d2a00 x1875453495664512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.686899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495664512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.686905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.686908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574905.686910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574905.686914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664640:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574905.686927:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574905.686933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff93a2460d1f80 x1875453495664640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.686943:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495664640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.686951:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.686955:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574905.686958:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495664640:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574905.686964:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495664768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574905.686984:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574905.687006:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff93a2460d2300 x1875453495664768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.687023:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495664768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.687043:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.687046:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574905.687050:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495664768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574906.646359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574906.646448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574907.670346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574907.670458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788574908.694429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574908.694430:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788574909.718364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574909.718457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574910.742326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574910.742389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574910.806391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.806404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574910.806410:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.806424:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3127 00000100:00080000:0.0:1788574910.806436:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.806446:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574910.806494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495664896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.806527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.806531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.806533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.806539:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3127 00000100:00080000:0.0:1788574910.806542:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.806546:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574910.806577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574910.806591:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495665024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.806602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.806606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.806608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.806613:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3127 00000100:00080000:0.0:1788574910.806618:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.806622:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574910.806627:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574910.806633:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495664896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574910.806652:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495665152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.806666:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.806670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.806672:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.806677:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3127 00000100:00080000:0.0:1788574910.806681:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.806685:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574910.806691:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574910.806703:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495665280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574910.806719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574910.806768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff93a2460d0380 x1875453495664896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.806783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495664896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.806791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.806794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574910.806797:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495664896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574910.806803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574910.806821:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574910.806829:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff93a2460d2680 x1875453495665024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.806837:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495665024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.806842:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.806845:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574910.806847:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574910.806882:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665152:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574910.806897:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574910.806905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff93a2460d1500 x1875453495665152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.806912:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495665152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.806926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.806928:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574910.806931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665152:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574910.806935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495665280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574910.806947:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574910.806954:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff93a2460d3480 x1875453495665280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.806961:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495665280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.806966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.806968:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574910.806970:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495665280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574911.766394:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574911.766461:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574912.790397:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574912.790484:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574913.814288:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574913.814409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574914.838360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574914.838448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574915.862355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574915.862453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574915.926375:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574915.926387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574915.926409:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574915.926419:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3132 00000100:00080000:0.0:1788574915.926430:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574915.926437:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574915.926472:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495665408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574915.926503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574915.926507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574915.926509:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574915.926513:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3132 00000100:00080000:0.0:1788574915.926517:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574915.926522:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574915.926532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574915.926547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495665536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574915.926558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574915.926562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574915.926564:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574915.926578:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3132 00000100:00080000:0.0:1788574915.926582:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574915.926589:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574915.926599:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574915.926599:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495665408:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574915.926631:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495665664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574915.926660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574915.926664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574915.926665:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574915.926668:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3132 00000100:00080000:0.0:1788574915.926685:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574915.926689:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574915.926694:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574915.926695:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788574915.926719:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495665792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574915.926764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff93a2460d0700 x1875453495665408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574915.926806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495665408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574915.926820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574915.926825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574915.926828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495665408:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574915.926836:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574915.926859:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574915.926870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff93a2460d0a80 x1875453495665536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574915.926879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495665536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574915.926885:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574915.926887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574915.926890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788574915.926917:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665664:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788574915.926931:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574915.926937:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff93a2460d0000 x1875453495665664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574915.926945:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495665664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574915.926960:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574915.926962:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574915.926965:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495665664:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788574915.926969:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495665792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788574915.926983:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788574915.926990:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff93a2460d3b80 x1875453495665792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574915.926998:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495665792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574915.927003:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574915.927005:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788574915.927008:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495665792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574916.886310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574916.886457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574917.910298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574917.910373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574918.934328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574918.934392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574919.958362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574919.958472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574920.982362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574920.982468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574921.046365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.046375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574921.046379:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.046396:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3137 00000100:00080000:1.0:1788574921.046408:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.046415:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574921.046462:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495665920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.046491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.046513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574921.046514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.046519:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3137 00000100:00080000:1.0:1788574921.046523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.046526:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574921.046537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574921.046544:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495665920:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574921.046552:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495666048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.046561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.046563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574921.046564:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.046568:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3137 00000100:00080000:1.0:1788574921.046570:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.046572:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574921.046575:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574921.046585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495666176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.046592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.046595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574921.046596:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.046599:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3137 00000100:00080000:1.0:1788574921.046601:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.046603:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574921.046611:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574921.046619:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495666304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574921.046628:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574921.046680:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff93a258da3800 x1875453495665920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.046696:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495665920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.046705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.046708:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574921.046711:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495665920:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574921.046736:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574921.046759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574921.046773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff93a258da1c00 x1875453495666048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.046785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495666048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.046795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.046798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574921.046801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574921.046805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666176:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574921.046823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574921.046842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff93a258da0700 x1875453495666176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.046849:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495666176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.046855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.046857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574921.046859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666176:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574921.046863:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495666304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574921.046876:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574921.046883:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff93a258da0a80 x1875453495666304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.046890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495666304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.046895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.046896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574921.046898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495666304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574922.006299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574922.006353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574923.030318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574923.030369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574924.054341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574924.054388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574925.078321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574925.078393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574926.102519:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574926.103826:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574926.166341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574926.166351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574926.166355:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574926.166364:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3142 00000100:00080000:1.0:1788574926.166374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574926.166380:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574926.166413:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495666432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574926.166437:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574926.166440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574926.166441:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574926.166445:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3142 00000100:00080000:1.0:1788574926.166447:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574926.166450:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574926.166458:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574926.166466:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495666560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574926.166474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574926.166477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574926.166478:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574926.166498:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3142 00000100:00080000:1.0:1788574926.166501:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574926.166505:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495666432:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574926.166506:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574926.166514:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574926.166523:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495666688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574926.166531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574926.166534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574926.166536:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574926.166539:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3142 00000100:00080000:1.0:1788574926.166542:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574926.166544:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574926.166547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574926.166556:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495666816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574926.166616:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574926.166675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666688:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574926.166705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495666816:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788574927.126499:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574927.126539:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574928.150969:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574928.151497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574929.174545:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788574929.174733:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574929.238580:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574929.238648:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574929.238672:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574929.238721:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574929.238730:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574929.238736:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574929.238802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff93a258da2680 x1875453495666688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.238836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495666688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.238850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.238858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574929.238869:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666688:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574929.238881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff93a258da0a80 x1875453495666816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.238902:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495666816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.238912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.238917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574929.238921:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495666816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574929.238930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff93a258da1f80 x1875453495666432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.238944:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495666432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.238983:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.238988:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574929.238992:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495666432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574929.239000:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff93a258da0e00 x1875453495666560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.239015:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495666560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.239023:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.239028:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574929.239031:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495666560:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788574930.198542:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574930.198718:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788574931.222352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574931.222459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574931.286363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574931.286378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574931.286384:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574931.286406:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3147 00000100:00080000:1.0:1788574931.286419:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574931.286426:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574931.286464:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495666944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574931.286493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574931.286512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574931.286514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574931.286518:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3147 00000100:00080000:1.0:1788574931.286521:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574931.286525:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574931.286536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574931.286547:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495667072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574931.286556:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495666944:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574931.286558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574931.286564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574931.286569:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574931.286573:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3147 00000100:00080000:1.0:1788574931.286575:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574931.286577:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574931.286581:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574931.286590:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495667200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574931.286597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574931.286599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574931.286601:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574931.286603:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3147 00000100:00080000:1.0:1788574931.286606:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574931.286608:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574931.286617:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574931.286626:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495667328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574931.286660:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574931.286718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff93a258da0700 x1875453495666944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.286749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495666944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.286761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.286767:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574931.286771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495666944:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574931.286805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574931.286868:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574931.286884:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff93a258da1c00 x1875453495667072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.286896:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495667072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.286906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.286910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574931.286913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574931.286919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667200:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574931.286942:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574931.286954:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff93a258da3800 x1875453495667200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.286966:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495667200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.286974:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.286978:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574931.286981:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667200:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574931.286987:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495667328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574931.287007:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574931.287019:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff93a258da1880 x1875453495667328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.287048:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495667328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.287056:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.287059:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574931.287062:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495667328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574932.246391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574932.246466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574933.270361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574933.270441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574934.294386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574934.294461:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574935.318359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574935.318451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574936.342340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574936.342433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574936.406384:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.406401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574936.406406:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.406421:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3152 00000100:00080000:1.0:1788574936.406436:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.406446:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574936.406498:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495667456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574936.406526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.406529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574936.406530:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.406534:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3152 00000100:00080000:1.0:1788574936.406537:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.406541:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574936.406551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574936.406565:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495667584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574936.406571:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495667456:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574936.406589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.406592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574936.406594:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.406598:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3152 00000100:00080000:1.0:1788574936.406618:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.406622:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574936.406628:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574936.406639:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495667712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574936.406648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.406651:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574936.406657:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574936.406658:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574936.406673:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3152 00000100:00080000:1.0:1788574936.406678:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.406683:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574936.406688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574936.406699:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495667840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574936.406722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff93a258da3100 x1875453495667456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.406751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495667456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.406764:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.406769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574936.406774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495667456:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574936.406803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574936.406832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574936.406847:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff93a258da2a00 x1875453495667584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.406862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495667584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.406915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.406920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574936.406924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574936.406932:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667712:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574936.406959:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574936.406969:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff93a258da3480 x1875453495667712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.406978:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495667712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.406985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.406990:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574936.406993:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495667712:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574936.407001:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495667840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574936.407023:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574936.407036:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff93a258da1180 x1875453495667840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.407045:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495667840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.407051:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.407053:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574936.407063:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495667840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574937.366381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574937.366454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574938.390377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788574938.390441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574939.414319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574939.414375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574940.438337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574940.438377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574941.462322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574941.462471:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574941.526321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.526328:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574941.526331:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.526339:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3157 00000100:00080000:1.0:1788574941.526347:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.526352:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574941.526382:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495667968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574941.526402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.526418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574941.526420:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.526423:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3157 00000100:00080000:1.0:1788574941.526425:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.526428:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574941.526436:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574941.526444:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495668096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574941.526452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.526455:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574941.526458:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574941.526459:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495667968:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574941.526463:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3157 00000100:00080000:1.0:1788574941.526467:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.526470:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574941.526474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574941.526480:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495668224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574941.526486:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.526489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574941.526490:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.526492:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3157 00000100:00080000:1.0:1788574941.526494:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.526495:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574941.526503:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574941.526511:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495668352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574941.526528:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574941.526579:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff93a258da2300 x1875453495667968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.526599:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495667968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.526609:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.526613:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574941.526617:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495667968:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574941.526638:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574941.526662:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574941.526675:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff93a258da0000 x1875453495668096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.526697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495668096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.526705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.526709:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574941.526712:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574941.526727:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668224:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574941.526747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574941.526757:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff93a258da0380 x1875453495668224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.526769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495668224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.526777:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.526781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574941.526785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668224:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574941.526791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495668352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574941.526812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574941.526823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff93a258da1180 x1875453495668352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.526835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495668352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.526843:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.526846:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574941.526849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495668352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574942.486297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574942.486333:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574943.510332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574943.510395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574944.534333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574944.534400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574945.558326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574945.558380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574946.582342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574946.582399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574946.646389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.646398:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574946.646402:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.646413:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3163 00000100:00080000:2.0:1788574946.646423:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.646430:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574946.646464:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495668480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.646495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.646497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574946.646499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.646502:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3163 00000100:00080000:2.0:1788574946.646523:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.646528:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574946.646536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574946.646547:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495668608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.646556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.646559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788574946.646560:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495668480:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788574946.646561:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.646578:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3163 00000100:00080000:2.0:1788574946.646581:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.646583:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574946.646585:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574946.646594:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495668736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.646601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.646604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574946.646605:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.646608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3163 00000100:00080000:2.0:1788574946.646610:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.646612:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574946.646615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574946.646623:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495668864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574946.646645:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574946.646697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff93a24940b480 x1875453495668480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.646716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495668480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.646728:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.646733:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574946.646737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495668480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574946.646763:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574946.646781:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574946.646789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff93a24940bb80 x1875453495668608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.646804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495668608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.646810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.646813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574946.646816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574946.646822:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668736:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574946.646843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574946.646853:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff93a24940b100 x1875453495668736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.646874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495668736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.646879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.646881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574946.646883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495668736:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574946.646888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495668864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574946.646902:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574946.646912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff93a249409500 x1875453495668864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.646923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495668864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.646930:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.646934:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574946.646937:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495668864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574947.606341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574947.606404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574948.630352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574948.630432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574949.654329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574949.654403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574950.678325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574950.678371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574951.702298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574951.702362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574951.766359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.766369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574951.766374:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.766385:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3168 00000100:00080000:2.0:1788574951.766396:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.766404:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574951.766447:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495668992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.766490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.766494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574951.766495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.766499:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3168 00000100:00080000:2.0:1788574951.766502:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.766506:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574951.766516:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574951.766530:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495669120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.766537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.766540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574951.766541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.766544:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3168 00000100:00080000:2.0:1788574951.766546:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.766548:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574951.766566:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495668992:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574951.766569:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574951.766595:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495669248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.766604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.766607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574951.766609:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.766613:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3168 00000100:00080000:2.0:1788574951.766616:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.766618:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574951.766621:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574951.766630:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495669376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574951.766657:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574951.766708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff93a249409880 x1875453495668992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.766735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495668992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.766744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.766748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574951.766751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495668992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574951.766757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574951.766788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574951.766802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff93a249408700 x1875453495669120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.766816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495669120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.766825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.766829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574951.766833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574951.766855:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669248:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574951.766872:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574951.766880:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff93a24940aa00 x1875453495669248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.766888:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495669248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.766902:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.766905:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574951.766907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669248:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574951.766912:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495669376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574951.766926:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574951.766933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff93a249408000 x1875453495669376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.766941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495669376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.766946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.766947:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574951.766949:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495669376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574952.726326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574952.726374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574953.750329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574953.750386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574954.774366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574954.774421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574955.798385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574955.798451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574956.822370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574956.822431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574956.886417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574956.886427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574956.886432:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574956.886442:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3173 00000100:00080000:1.0:1788574956.886453:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574956.886461:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574956.886500:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495669504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574956.886540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574956.886550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574956.886553:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574956.886559:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3173 00000100:00080000:1.0:1788574956.886565:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574956.886570:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574956.886574:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495669504:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574956.886582:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574956.886603:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495669632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574956.886615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574956.886619:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574956.886622:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574956.886627:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3173 00000800:00000100:3.0:1788574956.886631:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788574956.886651:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574956.886658:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574956.886665:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574956.886671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff93a258da3480 x1875453495669504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574956.886682:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495669760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574956.886688:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495669504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574956.886714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574956.886723:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574956.886728:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574956.886736:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574956.886737:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3173 00000100:00080000:1.0:1788574956.886745:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574956.886745:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:1.0:1788574956.886755:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574956.886755:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495669504:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788574956.886766:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574956.886782:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495669888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574956.886785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574956.886822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574956.886839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff93a258da2a00 x1875453495669632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574956.886853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495669632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.886859:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574956.886861:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574956.886863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574956.886868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669760:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574956.886885:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574956.886892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff93a258da3100 x1875453495669760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574956.886900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495669760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.886905:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574956.886907:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574956.886909:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495669760:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574956.886926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495669888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574956.886940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574956.886947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff93a258da1880 x1875453495669888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574956.886955:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495669888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.886960:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574956.886962:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574956.886964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495669888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574957.846344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574957.846455:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574958.870333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574958.870460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574959.894315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574959.894428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574960.918372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574960.918444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574961.942361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574961.942442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574962.006386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574962.006411:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574962.006418:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574962.006432:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3178 00000100:00080000:1.0:1788574962.006446:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574962.006455:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574962.006508:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495670016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574962.006543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574962.006567:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574962.006569:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574962.006574:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3178 00000100:00080000:1.0:1788574962.006579:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574962.006585:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574962.006595:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574962.006598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495670016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788574962.006608:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495670144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574962.006618:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574962.006620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574962.006622:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574962.006625:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3178 00000100:00080000:1.0:1788574962.006628:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574962.006630:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574962.006639:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574962.006647:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495670272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574962.006654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574962.006658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574962.006659:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574962.006661:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3178 00000100:00080000:1.0:1788574962.006664:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574962.006666:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788574962.006673:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788574962.006676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574962.006685:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495670400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574962.006727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff93a258da3800 x1875453495670016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.006769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495670016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.006778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.006781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574962.006784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495670016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574962.006814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574962.006832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574962.006842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff93a258da1c00 x1875453495670144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.006850:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495670144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.006856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.006858:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574962.006860:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574962.006863:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670272:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574962.006875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574962.006881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff93a258da0700 x1875453495670272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.006888:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495670272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.006893:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.006896:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574962.006898:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670272:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574962.006901:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495670400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574962.006927:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574962.006936:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff93a258da0e00 x1875453495670400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.006950:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495670400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.006963:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.006965:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574962.006967:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495670400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574962.966355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574962.966426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574963.990338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574963.990406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574965.014361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574965.014423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574966.038369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574966.038447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574967.062320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574967.062410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574967.126418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574967.126428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574967.126431:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574967.126440:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3183 00000100:00080000:0.0:1788574967.126449:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574967.126456:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574967.126491:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495670528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574967.126541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574967.126545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574967.126547:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574967.126552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3183 00000100:00080000:0.0:1788574967.126557:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574967.126561:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574967.126571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574967.126586:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495670656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574967.126597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574967.126601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574967.126603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574967.126607:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3183 00000100:00080000:0.0:1788574967.126610:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574967.126613:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574967.126619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574967.126629:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495670784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574967.126632:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495670528:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788574967.126639:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574967.126643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574967.126644:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574967.126647:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3183 00000100:00080000:0.0:1788574967.126655:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574967.126659:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574967.126663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574967.126673:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495670912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574967.126712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574967.126754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff93a2460d3b80 x1875453495670528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574967.126769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495670528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574967.126776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574967.126779:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574967.126782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495670528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574967.126788:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574967.126806:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574967.126814:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff93a2460d0000 x1875453495670656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574967.126830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495670656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574967.126836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574967.126838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574967.126840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574967.126860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670784:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574967.126881:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574967.126894:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff93a2460d0a80 x1875453495670784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574967.126908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495670784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574967.126917:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574967.126920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574967.126922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495670784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574967.126927:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495670912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574967.126942:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574967.126950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff93a2460d0700 x1875453495670912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574967.126957:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495670912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574967.126972:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574967.126974:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574967.126977:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495670912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574968.086345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574968.086405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574969.110306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574969.110409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574970.134324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574970.134384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574971.158357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574971.158423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574972.182399:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574972.182472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574972.246392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574972.246404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574972.246408:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574972.246420:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3188 00000100:00080000:2.0:1788574972.246438:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574972.246444:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574972.246479:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495671040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574972.246505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574972.246508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574972.246510:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574972.246513:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3188 00000100:00080000:2.0:1788574972.246530:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574972.246534:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574972.246543:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574972.246554:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495671040:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788574972.246560:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495671168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574972.246585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574972.246589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574972.246591:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574972.246595:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3188 00000100:00080000:2.0:1788574972.246599:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574972.246604:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574972.246609:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574972.246610:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788574972.246625:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495671296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574972.246635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574972.246638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574972.246639:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574972.246643:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3188 00000100:00080000:2.0:1788574972.246646:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574972.246650:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574972.246654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574972.246662:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495671424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574972.246703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff93a249408a80 x1875453495671040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574972.246721:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495671040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574972.246732:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574972.246738:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574972.246742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495671040:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574972.246766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574972.246787:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574972.246797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff93a249409180 x1875453495671168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574972.246806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495671168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574972.246818:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574972.246821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574972.246823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574972.246827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671296:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574972.246842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574972.246849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff93a24940ad80 x1875453495671296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574972.246857:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495671296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574972.246863:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574972.246865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574972.246867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671296:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574972.246870:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495671424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574972.246880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574972.246886:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff93a24940b800 x1875453495671424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574972.246892:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495671424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574972.246899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574972.246902:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574972.246913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495671424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574973.206382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574973.206433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574974.230400:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574974.230435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574975.254349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574975.254400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574976.278340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574976.278401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574977.302309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574977.302404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574977.366375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574977.366386:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574977.366391:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574977.366402:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3193 00000100:00080000:2.0:1788574977.366413:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574977.366421:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788574977.366464:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495671552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574977.366488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574977.366491:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574977.366492:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574977.366495:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3193 00000100:00080000:2.0:1788574977.366498:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574977.366500:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574977.366508:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574977.366517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495671680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574977.366524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574977.366527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574977.366528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574977.366531:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3193 00000100:00080000:2.0:1788574977.366532:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574977.366536:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574977.366557:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495671552:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788574977.366559:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574977.366574:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495671808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574977.366596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574977.366598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574977.366599:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574977.366604:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3193 00000100:00080000:2.0:1788574977.366607:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574977.366610:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788574977.366614:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574977.366626:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495671936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574977.366658:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574977.366712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff93a24940a680 x1875453495671552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574977.366741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495671552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574977.366755:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574977.366760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574977.366764:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495671552:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574977.366774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574977.366801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574977.366816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff93a249409c00 x1875453495671680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574977.366830:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495671680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574977.366840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574977.366844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574977.366862:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574977.366898:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671808:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574977.366923:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574977.366935:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff93a249408e00 x1875453495671808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574977.366949:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495671808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574977.366959:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574977.366963:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574977.366967:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495671808:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574977.366974:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495671936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574977.366995:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574977.367007:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff93a24940b800 x1875453495671936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574977.367021:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495671936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574977.367040:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574977.367044:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574977.367048:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495671936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574978.326382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574978.326445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574979.350319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574979.350385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574980.374297:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574980.374363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574981.398294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574981.398402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574982.422313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574982.422377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574982.486321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.486333:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574982.486338:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.486348:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3198 00000100:00080000:0.0:1788574982.486359:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.486366:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574982.486404:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495672064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574982.486457:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.486461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574982.486464:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.486470:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3198 00000100:00080000:0.0:1788574982.486476:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.486481:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574982.486492:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574982.486494:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495672064:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574982.486510:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495672192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574982.486534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.486539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574982.486542:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.486547:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3198 00000800:00000100:3.0:1788574982.486552:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788574982.486553:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.486560:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574982.486567:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574982.486581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495672320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574982.486594:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574982.486596:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff93a2460d3480 x1875453495672064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574982.486601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574982.486606:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.486614:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3198 00000100:00100000:3.0:1788574982.486620:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495672064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574982.486629:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574982.486632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574982.486637:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:3.0:1788574982.486638:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574982.486644:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495672064:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574982.486646:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574982.486662:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495672448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574982.486666:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574982.486691:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574982.486703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff93a2460d1500 x1875453495672192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574982.486712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495672192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574982.486718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574982.486720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574982.486723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574982.486726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672320:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574982.486759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574982.486767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff93a2460d2680 x1875453495672320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574982.486776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495672320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574982.486782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574982.486784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574982.486786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672320:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574982.486814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495672448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574982.486827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574982.486834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff93a2460d0380 x1875453495672448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574982.486842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495672448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574982.486848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574982.486851:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574982.486853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495672448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574983.446323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574983.446392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574984.470330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574984.470396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574985.494377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574985.494446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574986.518330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574986.518369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574987.542361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574987.542417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574987.606332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574987.606344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574987.606348:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574987.606358:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3204 00000100:00080000:0.0:1788574987.606368:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574987.606376:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574987.606412:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495672576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574987.606448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574987.606452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574987.606454:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574987.606459:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3204 00000100:00080000:0.0:1788574987.606463:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574987.606467:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574987.606478:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574987.606499:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495672576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788574987.606511:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495672704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574987.606535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574987.606539:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574987.606541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574987.606546:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3204 00000100:00080000:0.0:1788574987.606552:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574987.606557:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:3.0:1788574987.606562:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788574987.606566:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574987.606582:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495672832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574987.606593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574987.606597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574987.606598:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574987.606602:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff93a2460d2300 x1875453495672576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574987.606605:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3204 00000100:00080000:0.0:1788574987.606612:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574987.606619:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574987.606623:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574987.606626:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495672576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574987.606636:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574987.606640:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788574987.606644:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495672960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574987.606647:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495672576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574987.606675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574987.606693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574987.606701:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff93a2460d1f80 x1875453495672704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574987.606711:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495672704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574987.606720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574987.606724:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574987.606728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574987.606734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672832:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574987.606756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574987.606769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff93a2460d2a00 x1875453495672832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574987.606790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495672832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574987.606796:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574987.606798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574987.606800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495672832:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574987.606804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495672960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574987.606819:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574987.606826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff93a2460d1180 x1875453495672960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574987.606833:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495672960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574987.606838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574987.606839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574987.606841:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495672960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574988.566319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574988.566380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574989.590326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574989.590369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574990.614342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574990.614392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574991.638329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574991.638530:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574992.662565:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788574992.663742:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574992.726367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.726376:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574992.726380:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.726387:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3209 00000100:00080000:1.0:1788574992.726397:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.726403:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788574992.726440:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495673088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.726460:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.726463:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574992.726465:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.726468:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3209 00000100:00080000:1.0:1788574992.726470:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.726472:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574992.726480:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574992.726489:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495673216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.726497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.726502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788574992.726502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495673088:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788574992.726506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.726514:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3209 00000100:00080000:1.0:1788574992.726535:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.726538:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574992.726543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574992.726554:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495673344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.726562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.726564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574992.726566:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.726568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3209 00000100:00080000:1.0:1788574992.726571:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.726573:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788574992.726576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574992.726587:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495673472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574992.726612:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574992.726647:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673344:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574992.726664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495673472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788574993.686351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574993.686394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788574994.710400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574994.710449:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574995.734395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574995.735273:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788574995.798692:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788574995.798761:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574995.798787:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574995.798845:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574995.798857:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788574995.798873:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:3.0:1788574995.798913:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff93a258da0a80 x1875453495673216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.798948:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495673216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.798963:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.798970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574995.798974:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574995.798984:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff93a258da2680 x1875453495673344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.798995:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495673344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.799003:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.799006:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574995.799010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673344:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574995.799018:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff93a258da0e00 x1875453495673472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.799031:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495673472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.799065:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.799070:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574995.799074:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495673472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574995.799081:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff93a258da1f80 x1875453495673088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.799096:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495673088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.799103:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.799107:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574995.799111:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495673088:192.168.202.102@tcp:250:kworker.0 00010000:00010000:1.0:1788574996.758553:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574996.758744:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788574997.782307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574997.782344:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574997.846351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574997.846359:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574997.846363:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574997.846372:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3214 00000100:00080000:0.0:1788574997.846380:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574997.846387:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788574997.846417:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495673600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574997.846439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574997.846442:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574997.846443:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574997.846447:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3214 00000100:00080000:0.0:1788574997.846449:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574997.846451:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574997.846459:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574997.846480:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495673728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574997.846489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574997.846492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574997.846493:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574997.846497:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3214 00000100:00080000:0.0:1788574997.846500:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574997.846507:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788574997.846508:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495673600:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788574997.846525:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574997.846537:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495673856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574997.846546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574997.846548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574997.846549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574997.846553:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3214 00000100:00080000:0.0:1788574997.846558:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574997.846563:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788574997.846568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574997.846579:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495673984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574997.846590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574997.846640:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff93a2460d1180 x1875453495673600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574997.846663:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495673600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574997.846676:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574997.846681:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574997.846685:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495673600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788574997.846693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574997.846729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574997.846743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff93a2460d2a00 x1875453495673728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574997.846757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495673728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574997.846767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574997.846771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574997.846775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788574997.846828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673856:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788574997.846852:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574997.846864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff93a2460d1f80 x1875453495673856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574997.846877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495673856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574997.846886:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574997.846890:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574997.846893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495673856:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788574997.846900:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495673984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788574997.846919:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788574997.846930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff93a2460d2300 x1875453495673984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574997.846956:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495673984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574997.846966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574997.846970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788574997.846974:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495673984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788574998.806324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574998.806428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574999.830337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788574999.830357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575000.854339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575000.854371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575001.878323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575001.878364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575002.902348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575002.902367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575002.966405:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.966417:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575002.966439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.966461:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3219 00000100:00080000:1.0:1788575002.966472:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.966480:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575002.966512:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495674112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575002.966534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.966536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575002.966538:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.966541:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3219 00000100:00080000:1.0:1788575002.966544:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.966548:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575002.966556:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575002.966566:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495674112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575002.966568:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495674240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575002.966581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.966583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575002.966584:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.966587:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3219 00000100:00080000:1.0:1788575002.966588:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.966591:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575002.966595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575002.966602:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495674368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575002.966619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.966623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575002.966626:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.966630:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3219 00000100:00080000:1.0:1788575002.966635:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575002.966641:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575002.966645:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575002.966652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575002.966666:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495674496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575002.966705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff93a258da0700 x1875453495674112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575002.966726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495674112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575002.966734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575002.966737:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575002.966740:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495674112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575002.966759:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575002.966778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575002.966787:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff93a258da1c00 x1875453495674240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575002.966796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495674240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575002.966813:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575002.966816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575002.966818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575002.966824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674368:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788575002.966860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575002.966873:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff93a258da3800 x1875453495674368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575002.966885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495674368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575002.966894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575002.966898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575002.966900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674368:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788575002.966905:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495674496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575002.966920:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575002.966928:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff93a258da1880 x1875453495674496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575002.966935:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495674496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575002.966941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575002.966943:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575002.966950:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495674496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575003.926358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575003.926391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575004.950387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575004.950422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575005.974337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575005.974366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575006.998341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575006.998373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575008.022310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575008.022349:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575008.086391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575008.086408:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575008.086413:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575008.086428:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3224 00000100:00080000:1.0:1788575008.086442:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575008.086450:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575008.086503:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495674624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575008.086549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575008.086561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575008.086581:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575008.086588:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3224 00000100:00080000:1.0:1788575008.086593:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575008.086598:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575008.086612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575008.086630:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495674752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575008.086645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575008.086654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788575008.086656:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495674624:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575008.086657:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575008.086671:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3224 00000100:00080000:1.0:1788575008.086676:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575008.086681:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575008.086687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575008.086703:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495674880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575008.086715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575008.086719:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575008.086722:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575008.086727:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3224 00000100:00080000:1.0:1788575008.086734:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575008.086739:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575008.086746:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788575008.086747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575008.086777:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495675008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575008.086816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff93a258da3100 x1875453495674624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575008.086842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495674624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575008.086857:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575008.086863:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575008.086868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495674624:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575008.086910:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575008.086937:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575008.086952:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff93a258da2a00 x1875453495674752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575008.086967:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495674752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575008.086977:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575008.086981:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575008.086985:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575008.086992:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674880:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788575008.087016:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575008.087060:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff93a258da3480 x1875453495674880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575008.087074:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495674880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575008.087083:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575008.087087:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575008.087091:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495674880:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788575008.087098:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495675008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575008.087121:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575008.087134:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff93a258da1180 x1875453495675008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575008.087146:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495675008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575008.087155:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575008.087159:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575008.087162:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495675008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575009.046343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575009.046368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575010.070375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575010.070422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575011.094388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575011.094454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575012.118359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575012.118422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575013.142374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575013.142462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575013.206427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575013.206440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575013.206446:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575013.206458:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3229 00000100:00080000:0.0:1788575013.206471:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575013.206479:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575013.206533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495675136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575013.206563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575013.206598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575013.206601:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575013.206608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3229 00000100:00080000:0.0:1788575013.206613:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575013.206617:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575013.206628:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575013.206661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495675136:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575013.206679:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495675264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575013.206700:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575013.206705:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575013.206707:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575013.206715:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3229 00000100:00080000:0.0:1788575013.206720:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575013.206723:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575013.206731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575013.206743:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495675392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575013.206759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575013.206773:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575013.206779:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575013.206782:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575013.206792:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3229 00000100:00080000:0.0:1788575013.206798:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575013.206804:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575013.206820:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575013.206824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff93a2460d0380 x1875453495675136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575013.206843:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495675520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575013.206853:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495675136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575013.206882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575013.206888:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575013.206893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495675136:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575013.206922:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575013.206949:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575013.206962:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff93a2460d2680 x1875453495675264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575013.206974:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495675264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575013.206983:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575013.206986:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575013.206990:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575013.206997:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675392:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788575013.207052:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575013.207064:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff93a2460d1500 x1875453495675392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575013.207076:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495675392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575013.207097:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575013.207101:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575013.207105:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675392:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788575013.207118:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495675520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575013.207141:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575013.207152:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff93a2460d3480 x1875453495675520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575013.207164:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495675520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575013.207171:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575013.207175:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575013.207178:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495675520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575014.166341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575014.166399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575015.190394:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575015.190462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575016.214371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575016.214406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575017.238344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575017.238400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575018.262314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575018.262388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575018.326339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575018.326348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575018.326352:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575018.326361:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3234 00000100:00080000:0.0:1788575018.326370:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575018.326376:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575018.326411:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495675648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575018.326429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575018.326432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575018.326434:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575018.326437:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3234 00000100:00080000:0.0:1788575018.326439:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575018.326442:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575018.326450:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575018.326456:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495675648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575018.326462:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495675776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575018.326495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575018.326497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575018.326499:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575018.326503:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3234 00000100:00080000:0.0:1788575018.326506:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575018.326509:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575018.326513:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575018.326526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495675904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575018.326538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575018.326542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575018.326545:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575018.326549:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3234 00000800:00000100:3.0:1788575018.326556:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575018.326561:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575018.326570:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575018.326578:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575018.326598:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495676032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575018.326622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff93a2460d0700 x1875453495675648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.326647:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495675648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.326657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.326660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575018.326663:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495675648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575018.326689:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575018.326708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575018.326717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff93a2460d0a80 x1875453495675776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.326738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495675776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.326745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.326747:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575018.326749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575018.326754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675904:192.168.202.102@tcp:8:kworker.0 00000800:00000100:3.0:1788575018.326770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575018.326777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff93a2460d0000 x1875453495675904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.326790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495675904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.326806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.326810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575018.326813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495675904:192.168.202.102@tcp:8:kworker.0 00000100:00100000:3.0:1788575018.326821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495676032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:3.0:1788575018.326851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575018.326866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff93a2460d3b80 x1875453495676032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.326880:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495676032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.326898:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.326901:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575018.326904:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495676032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575019.286334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575019.286404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575020.310305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575020.310364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575021.334343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575021.334401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575022.358336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575022.358393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575023.382350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575023.382423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575023.446366:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.446378:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575023.446383:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.446394:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3239 00000100:00080000:3.0:1788575023.446413:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.446420:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575023.446471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495676160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575023.446512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.446517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575023.446519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.446524:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3239 00000100:00080000:3.0:1788575023.446529:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.446536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575023.446548:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575023.446564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495676288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575023.446576:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495676160:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575023.446580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.446592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575023.446615:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.446621:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3239 00000100:00080000:3.0:1788575023.446624:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.446628:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575023.446633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575023.446645:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495676416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575023.446654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.446657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575023.446659:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.446666:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3239 00000800:00000100:1.0:1788575023.446669:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575023.446672:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.446678:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575023.446682:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575023.446693:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495676544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575023.446721:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff93a25064e680 x1875453495676160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575023.446742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495676160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575023.446754:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575023.446759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575023.446763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495676160:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575023.446793:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575023.446820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575023.446834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff93a25064ea00 x1875453495676288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575023.446847:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495676288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575023.446856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575023.446860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575023.446863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575023.446870:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676416:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575023.446905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575023.446914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff93a25064d880 x1875453495676416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575023.446922:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495676416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575023.446928:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575023.446930:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575023.446932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676416:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575023.446937:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495676544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575023.446950:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575023.446956:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff93a25064dc00 x1875453495676544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575023.446963:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495676544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575023.446968:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575023.446970:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575023.446972:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495676544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575024.406350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575024.406424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575025.430328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575025.430352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575026.454375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575026.454417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575027.478315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575027.478420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575028.502309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575028.502381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575028.566318:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575028.566330:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575028.566334:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575028.566342:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3245 00000100:00080000:2.0:1788575028.566380:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575028.566387:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575028.566416:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495676672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575028.566440:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575028.566443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575028.566445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575028.566448:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3245 00000100:00080000:2.0:1788575028.566450:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575028.566453:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575028.566460:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575028.566468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495676800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575028.566475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575028.566477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575028.566479:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575028.566482:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3245 00000100:00080000:2.0:1788575028.566484:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575028.566486:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575028.566489:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575028.566496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495676928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575028.566502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495676672:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575028.566506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575028.566525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575028.566527:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575028.566531:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3245 00000100:00080000:2.0:1788575028.566534:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575028.566538:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575028.566543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575028.566552:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495677056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575028.566570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575028.566610:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff93a24940ad80 x1875453495676672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.566642:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495676672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.566660:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.566664:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575028.566669:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495676672:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575028.566676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575028.566700:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575028.566714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff93a249409180 x1875453495676800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.566727:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495676800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.566735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.566738:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575028.566741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575028.566766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676928:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575028.566788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575028.566800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff93a249408a80 x1875453495676928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.566812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495676928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.566833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.566836:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575028.566838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495676928:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575028.566843:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495677056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575028.566859:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575028.566867:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff93a249408000 x1875453495677056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.566875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495677056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.566880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.566882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575028.566885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495677056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575029.526319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575029.526431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575030.550323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575030.550400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575031.574306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575031.574382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575032.598316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575032.598378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575033.622331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575033.622386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575033.686406:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.686419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575033.686425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.686438:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3250 00000100:00080000:3.0:1788575033.686453:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.686461:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575033.686526:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495677184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.686589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.686593:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.686595:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.686600:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3250 00000100:00080000:3.0:1788575033.686605:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.686610:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575033.686622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.686641:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495677312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.686654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.686657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.686660:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575033.686671:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495677184:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575033.686673:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3250 00000100:00080000:3.0:1788575033.686681:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.686691:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575033.686697:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.686713:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495677440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.686725:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.686728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.686731:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.686736:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3250 00000800:00000100:1.0:1788575033.686757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575033.686763:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.686773:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575033.686781:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.686797:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495677568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575033.686816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff93a25064ed80 x1875453495677184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575033.686842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495677184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575033.686856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575033.686861:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575033.686866:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495677184:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575033.686924:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575033.686954:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575033.686968:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff93a25064dc00 x1875453495677312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575033.686983:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495677312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575033.686992:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575033.686996:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575033.687000:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575033.687007:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677440:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575033.687030:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575033.687043:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff93a25064d880 x1875453495677440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575033.687056:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495677440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575033.687074:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575033.687079:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575033.687082:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575033.687090:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495677568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575033.687113:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575033.687125:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff93a25064ea00 x1875453495677568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575033.687146:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495677568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575033.687154:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575033.687158:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575033.687161:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495677568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575034.646363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575034.646441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575035.670341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575035.670409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575036.694311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575036.694373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575037.718309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575037.718361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575038.742308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575038.742366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575038.806361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575038.806371:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575038.806375:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575038.806385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3255 00000100:00080000:2.0:1788575038.806395:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575038.806401:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575038.806433:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495677696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575038.806476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575038.806480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575038.806481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575038.806485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3255 00000100:00080000:2.0:1788575038.806489:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575038.806493:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575038.806503:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575038.806512:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495677696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575038.806518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495677824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575038.806528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575038.806531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575038.806533:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575038.806538:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3255 00000100:00080000:2.0:1788575038.806542:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575038.806546:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575038.806551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575038.806561:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495677952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575038.806571:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575038.806576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575038.806580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575038.806582:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575038.806587:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3255 00000100:00080000:2.0:1788575038.806600:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575038.806619:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575038.806622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff93a24940aa00 x1875453495677696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575038.806626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575038.806639:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495677696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.806644:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495678080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575038.806650:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575038.806657:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575038.806662:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495677696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575038.806682:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575038.806698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575038.806705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff93a249408700 x1875453495677824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575038.806712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495677824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575038.806726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575038.806728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575038.806730:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575038.806734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677952:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575038.806746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575038.806752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff93a249409880 x1875453495677952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575038.806767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495677952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575038.806771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575038.806773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575038.806775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495677952:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575038.806778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495678080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575038.806788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575038.806793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff93a249409500 x1875453495678080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575038.806799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495678080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575038.806803:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575038.806805:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575038.806811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495678080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575039.766282:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575039.766334:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575040.790312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575040.790405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575041.814351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575041.814404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575042.838357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575042.838415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575043.862326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575043.862372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575043.926338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575043.926350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575043.926354:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575043.926364:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3260 00000100:00080000:2.0:1788575043.926374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575043.926381:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575043.926423:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495678208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575043.926453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575043.926457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575043.926459:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575043.926464:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3260 00000100:00080000:2.0:1788575043.926468:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575043.926472:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575043.926481:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575043.926505:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495678208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575043.926512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495678336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575043.926532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575043.926535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575043.926538:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575043.926543:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3260 00000100:00080000:2.0:1788575043.926548:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575043.926552:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575043.926555:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:2.0:1788575043.926561:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575043.926576:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495678464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575043.926587:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575043.926591:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575043.926593:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575043.926599:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3260 00000100:00100000:1.0:1788575043.926600:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff93a24940b100 x1875453495678208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575043.926603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575043.926611:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575043.926619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575043.926629:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495678208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575043.926634:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495678592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575043.926657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575043.926664:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575043.926668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495678208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575043.926688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575043.926705:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575043.926715:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff93a24940bb80 x1875453495678336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575043.926741:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495678336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575043.926750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575043.926753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575043.926757:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575043.926763:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678464:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575043.926784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575043.926803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff93a24940b480 x1875453495678464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575043.926814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495678464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575043.926820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575043.926822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575043.926824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678464:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575043.926828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495678592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575043.926841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575043.926847:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff93a249409500 x1875453495678592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575043.926854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495678592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575043.926858:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575043.926860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575043.926861:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495678592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575044.886322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575044.886372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575045.910336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575045.910416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575046.934353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575046.934385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575047.958356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575047.958430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575048.982369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575048.982446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575049.046401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575049.046415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575049.046420:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575049.046433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3265 00000100:00080000:0.0:1788575049.046447:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575049.046455:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575049.046524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495678720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575049.046567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575049.046572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575049.046574:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575049.046579:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3265 00000100:00080000:0.0:1788575049.046584:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575049.046589:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575049.046601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575049.046615:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495678720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575049.046617:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495678848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575049.046631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575049.046635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575049.046637:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575049.046642:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3265 00000100:00080000:0.0:1788575049.046646:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575049.046651:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575049.046656:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575049.046669:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495678976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575049.046679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575049.046683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575049.046685:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575049.046692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575049.046699:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3265 00000100:00080000:0.0:1788575049.046704:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575049.046710:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575049.046717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575049.046731:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495679104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575049.046741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff93a2460d3b80 x1875453495678720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575049.046757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495678720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575049.046775:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575049.046780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575049.046784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495678720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575049.046812:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575049.046829:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575049.046865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff93a2460d0000 x1875453495678848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575049.046873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495678848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575049.046883:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575049.046886:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575049.046890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575049.046895:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678976:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575049.046916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575049.046927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff93a2460d0a80 x1875453495678976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575049.046937:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495678976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575049.046942:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575049.046944:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575049.046946:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495678976:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575049.046955:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495679104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575049.046969:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575049.046976:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff93a2460d0700 x1875453495679104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575049.046983:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495679104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575049.046988:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575049.046990:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575049.046993:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495679104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575050.006324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575050.006385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575051.030355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575051.030434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575052.054362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575052.054423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575053.078360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575053.078435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575054.102377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575054.102427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575054.166401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575054.166412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575054.166417:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575054.166427:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3270 00000100:00080000:3.0:1788575054.166437:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575054.166445:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575054.166485:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495679232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575054.166518:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575054.166522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575054.166524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575054.166528:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3270 00000100:00080000:3.0:1788575054.166532:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575054.166537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575054.166547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575054.166560:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495679360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575054.166568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575054.166584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575054.166586:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575054.166590:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495679232:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575054.166592:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3270 00000100:00080000:3.0:1788575054.166597:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575054.166602:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575054.166606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575054.166614:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495679488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575054.166623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575054.166626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575054.166627:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575054.166630:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3270 00000100:00080000:3.0:1788575054.166632:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575054.166634:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575054.166637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575054.166644:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495679616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575054.166666:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575054.166711:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff93a25064e680 x1875453495679232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.166731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495679232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.166742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.166746:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575054.166750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495679232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575054.166757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495679360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575054.166779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575054.166791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff93a25064fb80 x1875453495679360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.166812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495679360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.166821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.166825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575054.166828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495679360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575054.166856:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495679488:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575054.166880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575054.166893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff93a25064f480 x1875453495679488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.166906:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495679488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.166912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.166914:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575054.166916:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495679488:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575054.166921:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495679616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575054.166936:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575054.166942:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff93a25064c000 x1875453495679616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.166950:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495679616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.167006:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.167011:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575054.167015:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495679616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575055.126368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575055.126423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575056.150364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575056.150431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575057.174335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575057.174413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575058.198391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575058.198485:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575059.222407:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575059.222439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575059.286383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.286393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575059.286398:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.286409:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3275 00000100:00080000:2.0:1788575059.286420:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.286427:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575059.286470:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495679744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.286494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.286496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575059.286498:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.286501:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3275 00000100:00080000:2.0:1788575059.286503:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.286505:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575059.286514:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575059.286524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495679872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.286553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.286555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575059.286557:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.286560:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3275 00000100:00080000:2.0:1788575059.286564:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.286571:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575059.286573:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495679744:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788575059.286575:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575059.286596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495680000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.286608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.286611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575059.286612:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.286616:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3275 00000100:00080000:2.0:1788575059.286618:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.286621:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575059.286626:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575059.286640:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495680128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575059.286683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495679872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575059.286716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680000:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575059.286735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495680128:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788575060.246441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575060.246689:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575061.270491:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575061.270809:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575062.294646:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575062.294688:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788575062.358589:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788575062.358633:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575062.358655:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575062.358709:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575062.358730:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575062.358742:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788575062.358751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff93a24940aa00 x1875453495680000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575062.358782:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495680000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575062.358795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575062.358807:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575062.358811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680000:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575062.358822:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff93a249408000 x1875453495680128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575062.358862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495680128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575062.358873:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575062.358878:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575062.358882:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495680128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575062.358890:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff93a249409880 x1875453495679744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575062.358905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495679744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575062.358915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575062.358918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575062.358923:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495679744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575062.358930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff93a249408700 x1875453495679872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575062.358943:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495679872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575062.358952:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575062.358956:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575062.358959:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495679872:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788575063.318434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575063.318929:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575064.342337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575064.342367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575064.406417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.406433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575064.406438:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.406452:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3280 00000100:00080000:2.0:1788575064.406466:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.406476:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575064.406526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495680256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.406568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.406573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575064.406575:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.406580:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3280 00000100:00080000:2.0:1788575064.406585:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.406589:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575064.406601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575064.406617:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495680384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.406631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.406635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575064.406637:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575064.406644:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495680256:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575064.406646:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3280 00000100:00080000:2.0:1788575064.406650:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.406654:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575064.406660:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575064.406686:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495680512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.406699:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.406704:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575064.406706:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.406710:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3280 00000100:00080000:2.0:1788575064.406715:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.406720:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575064.406726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575064.406739:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495680640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575064.406802:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575064.406889:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff93a249408a80 x1875453495680256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.406914:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495680256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.406926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.406931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575064.406935:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495680256:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575064.407276:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575064.407312:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575064.407326:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff93a249409180 x1875453495680384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.407339:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495680384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.407348:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.407358:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575064.407362:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575064.407370:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680512:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575064.407393:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575064.407404:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff93a24940ad80 x1875453495680512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.407431:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495680512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.407445:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.407448:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575064.407451:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680512:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575064.407458:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495680640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575064.407481:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575064.407492:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff93a24940b800 x1875453495680640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.407503:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495680640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.407511:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.407515:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575064.407518:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495680640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575065.366370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575065.366406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575066.390357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575066.390374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575067.414404:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575067.414425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575068.438402:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575068.438432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575069.462352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575069.462374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575069.526336:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575069.526381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575069.526387:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575069.526402:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3285 00000100:00080000:3.0:1788575069.526415:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575069.526424:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575069.526478:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495680768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575069.526513:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575069.526516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575069.526517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575069.526521:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3285 00000100:00080000:3.0:1788575069.526524:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575069.526528:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575069.526540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575069.526558:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495680896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575069.526570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575069.526574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575069.526576:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575069.526581:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3285 00000100:00080000:3.0:1788575069.526584:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575069.526592:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575069.526597:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495680768:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788575069.526599:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575069.526617:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495681024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575069.526639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575069.526643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575069.526645:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575069.526651:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3285 00000100:00080000:3.0:1788575069.526657:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575069.526671:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575069.526680:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575069.526689:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575069.526699:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495681152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575069.526753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff93a25064d180 x1875453495680768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575069.526775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495680768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575069.526807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575069.526812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575069.526816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495680768:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575069.526844:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575069.526871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575069.526885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff93a25064f100 x1875453495680896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575069.526900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495680896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575069.526927:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575069.526931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575069.526935:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495680896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575069.526943:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681024:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575069.526966:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575069.526978:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff93a25064ce00 x1875453495681024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575069.526991:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495681024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575069.527000:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575069.527004:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575069.527008:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681024:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575069.527022:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495681152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575069.527048:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575069.527078:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff93a25064e300 x1875453495681152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575069.527091:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495681152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575069.527101:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575069.527105:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575069.527109:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495681152:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788575070.486357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575070.486361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788575071.510381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575071.510418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575072.534341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575072.534434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575073.558364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575073.558415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575074.582334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575074.582387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575074.646319:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.646328:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575074.646332:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.646342:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3291 00000100:00080000:0.0:1788575074.646352:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.646361:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575074.646399:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495681280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.646437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.646441:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575074.646455:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.646458:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3291 00000100:00080000:0.0:1788575074.646462:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.646466:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575074.646475:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575074.646486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495681408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.646494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.646497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575074.646499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.646502:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3291 00000100:00080000:0.0:1788575074.646504:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.646506:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575074.646509:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575074.646516:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495681536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.646525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.646528:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575074.646530:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.646539:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3291 00000100:00100000:1.0:1788575074.646542:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495681280:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575074.646543:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.646546:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575074.646552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575074.646568:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495681664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575074.646659:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575074.646706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff93a2460d3480 x1875453495681280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.646730:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495681280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.646740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.646744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575074.646747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495681280:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575074.646754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575074.646772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575074.646781:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff93a2460d1500 x1875453495681408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.646789:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495681408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.646795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.646797:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575074.646800:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575074.646824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681536:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575074.646837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575074.646843:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff93a2460d2680 x1875453495681536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.646862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495681536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.646868:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.646870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575074.646872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681536:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575074.646876:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495681664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575074.646890:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575074.646896:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff93a2460d0380 x1875453495681664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.646904:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495681664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.646909:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.646910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575074.646912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495681664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575075.606368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575075.606392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575076.630358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575076.630379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575077.654318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575077.654326:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575078.678302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575078.678345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575079.702340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575079.702413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575079.766383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575079.766397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575079.766402:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575079.766416:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3296 00000100:00080000:2.0:1788575079.766429:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575079.766438:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575079.766482:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495681792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575079.766548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575079.766551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575079.766552:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575079.766556:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3296 00000100:00080000:2.0:1788575079.766559:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575079.766563:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575079.766573:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575079.766585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495681920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575079.766593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575079.766595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575079.766597:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575079.766600:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3296 00000100:00080000:2.0:1788575079.766602:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575079.766606:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495681792:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575079.766607:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575079.766611:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575079.766619:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495682048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575079.766625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575079.766627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575079.766628:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575079.766631:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3296 00000100:00080000:2.0:1788575079.766638:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575079.766641:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575079.766645:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575079.766656:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495682176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575079.766689:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575079.766739:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff93a249408e00 x1875453495681792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.766773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495681792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.766784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.766788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575079.766793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495681792:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575079.766819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575079.766839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575079.766851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff93a249409c00 x1875453495681920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.766863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495681920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.766872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.766875:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575079.766878:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495681920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575079.766883:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682048:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575079.766900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575079.766909:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff93a24940a680 x1875453495682048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.766921:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495682048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.766939:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.766943:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575079.766946:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682048:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575079.766951:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495682176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575079.766969:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575079.766978:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff93a24940b800 x1875453495682176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.766987:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495682176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.766995:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.766997:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575079.767000:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495682176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575080.726332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575080.726411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575081.750324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575081.750384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575082.774381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575082.774387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788575083.798321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575083.798435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575084.822397:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575084.822456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575084.886351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575084.886362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575084.886366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575084.886379:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3301 00000100:00080000:3.0:1788575084.886393:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575084.886401:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575084.886448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495682304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575084.886477:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575084.886480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575084.886482:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575084.886485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3301 00000100:00080000:3.0:1788575084.886487:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575084.886490:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575084.886520:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575084.886531:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495682432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575084.886538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495682304:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575084.886542:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575084.886546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575084.886547:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575084.886551:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3301 00000100:00080000:3.0:1788575084.886553:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575084.886555:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575084.886559:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575084.886566:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495682560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575084.886573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575084.886576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575084.886578:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575084.886591:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3301 00000100:00080000:3.0:1788575084.886595:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575084.886599:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575084.886603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575084.886615:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495682688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575084.886623:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575084.886671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff93a25064d500 x1875453495682304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575084.886686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495682304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575084.886694:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575084.886703:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575084.886706:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495682304:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575084.886731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575084.886747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575084.886755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff93a25064c700 x1875453495682432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575084.886762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495682432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575084.886767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575084.886769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575084.886771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575084.886774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682560:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575084.886784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575084.886789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff93a25064ca80 x1875453495682560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575084.886795:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495682560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575084.886799:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575084.886801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575084.886802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682560:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575084.886805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495682688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575084.886822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575084.886827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff93a25064c380 x1875453495682688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575084.886834:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495682688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575084.886838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575084.886840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575084.886841:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495682688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575085.846333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575085.846392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575086.870359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575086.870433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575087.894366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575087.894433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575088.918392:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575088.918447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575089.942345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575089.942400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575090.006363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.006374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575090.006379:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.006390:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3306 00000100:00080000:0.0:1788575090.006400:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.006408:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575090.006453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495682816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575090.006479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.006482:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575090.006483:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.006486:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3306 00000100:00080000:0.0:1788575090.006507:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.006511:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575090.006521:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575090.006531:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495682944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575090.006539:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495682816:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575090.006541:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.006548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575090.006550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.006554:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3306 00000100:00080000:0.0:1788575090.006556:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.006558:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575090.006561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575090.006569:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495683072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575090.006575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.006578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575090.006579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.006581:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3306 00000100:00080000:0.0:1788575090.006583:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.006585:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575090.006587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575090.006594:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495683200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575090.006627:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575090.006688:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff93a2460d2300 x1875453495682816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.006707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495682816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.006718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.006722:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575090.006725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495682816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575090.006752:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575090.006776:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575090.006788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff93a2460d1f80 x1875453495682944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.006813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495682944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.006820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.006823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575090.006826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495682944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575090.006832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683072:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575090.006853:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575090.006864:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff93a2460d2a00 x1875453495683072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.006874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495683072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.006881:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.006884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575090.006886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683072:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575090.006891:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495683200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575090.006909:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575090.006919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff93a2460d1180 x1875453495683200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.006930:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495683200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.006947:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.006950:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575090.006953:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495683200:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788575090.966505:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575090.966576:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575091.990353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575091.990416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575093.014368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575093.014434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575094.038375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575094.038441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575095.062356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575095.062398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575095.126370:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575095.126381:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575095.126386:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575095.126397:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3311 00000100:00080000:0.0:1788575095.126408:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575095.126416:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575095.126460:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495683328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575095.126486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575095.126488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575095.126490:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575095.126493:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3311 00000100:00080000:0.0:1788575095.126495:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575095.126497:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575095.126507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575095.126519:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495683456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575095.126529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575095.126537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575095.126547:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495683328:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575095.126558:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575095.126564:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3311 00000100:00080000:0.0:1788575095.126568:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575095.126572:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575095.126578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575095.126590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495683584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575095.126599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575095.126601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575095.126603:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575095.126606:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3311 00000100:00080000:0.0:1788575095.126616:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575095.126618:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575095.126621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575095.126631:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495683712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575095.126631:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575095.126691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff93a2460d1180 x1875453495683328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.126723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495683328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.126735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.126740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575095.126743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495683328:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575095.126748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575095.126788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575095.126798:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff93a2460d2a00 x1875453495683456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.126809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495683456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.126817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.126820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575095.126824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575095.126858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683584:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575095.126874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575095.126881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff93a2460d1f80 x1875453495683584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.126888:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495683584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.126893:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.126895:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575095.126897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683584:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575095.126900:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495683712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575095.126909:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575095.126914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff93a2460d2300 x1875453495683712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.126921:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495683712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.126925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.126927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575095.126928:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495683712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575096.086330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575096.086392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575097.110385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575097.110445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575098.134290:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575098.134359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575099.158335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575099.158402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575100.182329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575100.182387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575100.246410:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.246426:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575100.246461:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.246475:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3316 00000100:00080000:3.0:1788575100.246488:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.246496:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575100.246539:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495683840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.246576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.246580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575100.246582:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.246587:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3316 00000100:00080000:3.0:1788575100.246590:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.246595:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575100.246607:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575100.246623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495683968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.246634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575100.246640:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495683840:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575100.246640:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575100.246643:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.246652:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3316 00000100:00080000:3.0:1788575100.246656:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.246660:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575100.246665:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575100.246677:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495684096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.246700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.246703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575100.246705:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.246709:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3316 00000100:00080000:3.0:1788575100.246714:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.246718:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575100.246724:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575100.246726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575100.246738:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495684224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575100.246786:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff93a25064df80 x1875453495683840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.246814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495683840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.246825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.246830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575100.246834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495683840:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575100.246877:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575100.246900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575100.246912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff93a25064c380 x1875453495683968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.246924:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495683968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.246932:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.246935:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575100.246937:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495683968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575100.246942:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684096:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575100.246959:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575100.246968:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff93a25064ca80 x1875453495684096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.246979:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495684096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.246996:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.246999:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575100.247002:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684096:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575100.247007:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495684224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575100.247038:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575100.247047:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff93a25064c700 x1875453495684224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.247056:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495684224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.247063:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.247065:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575100.247068:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495684224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575101.206308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575101.206409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575102.230317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575102.230379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575103.254319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575103.254418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575104.278374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575104.278437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575105.302349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575105.302414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575105.366395:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575105.366409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575105.366415:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575105.366428:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3321 00000100:00080000:0.0:1788575105.366441:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575105.366450:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575105.366497:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495684352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575105.366531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575105.366535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575105.366536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575105.366541:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3321 00000100:00080000:0.0:1788575105.366545:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575105.366549:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575105.366561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575105.366575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495684480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575105.366581:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495684352:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575105.366593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575105.366597:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575105.366599:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575105.366604:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3321 00000100:00080000:0.0:1788575105.366608:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575105.366612:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575105.366618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575105.366651:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495684608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575105.366654:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575105.366666:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575105.366670:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575105.366673:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575105.366678:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3321 00000100:00080000:0.0:1788575105.366682:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575105.366687:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575105.366692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575105.366704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff93a2460d0380 x1875453495684352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575105.366708:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495684736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575105.366729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495684352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575105.366739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575105.366742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575105.366753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495684352:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575105.366775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575105.366792:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575105.366800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff93a2460d2680 x1875453495684480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575105.366807:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495684480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575105.366812:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575105.366814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575105.366816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575105.366819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684608:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575105.366831:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575105.366836:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff93a2460d1500 x1875453495684608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575105.366843:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495684608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575105.366847:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575105.366848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575105.366850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684608:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575105.366853:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495684736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575105.366866:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575105.366885:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff93a2460d3480 x1875453495684736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575105.366896:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495684736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575105.366904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575105.366907:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575105.366910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495684736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575106.326375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575106.326442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575107.350322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575107.350377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575108.374356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575108.374399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575109.398317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575109.398413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575110.422337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575110.422386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575110.486330:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.486337:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575110.486340:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.486349:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3326 00000100:00080000:3.0:1788575110.486358:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.486364:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575110.486398:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495684864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.486421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.486425:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575110.486427:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.486431:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3326 00000100:00080000:3.0:1788575110.486435:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.486440:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575110.486455:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495684864:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788575110.486464:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575110.486482:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495684992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.486501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.486503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575110.486505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575110.486509:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575110.486512:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3326 00000100:00080000:3.0:1788575110.486518:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.486523:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575110.486529:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575110.486543:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff93a25064d500 x1875453495684864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575110.486549:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495685120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575110.486563:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495684864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575110.486566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.486576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575110.486577:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575110.486579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575110.486582:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575110.486588:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495684864:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575110.486590:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3326 00000100:00080000:3.0:1788575110.486596:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.486601:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575110.486607:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575110.486611:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575110.486623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495685248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575110.486631:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575110.486643:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff93a25064e300 x1875453495684992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.486652:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495684992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.486669:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.486671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575110.486674:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495684992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575110.486677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495685120:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575110.486690:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575110.486696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff93a25064ce00 x1875453495685120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.486707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495685120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.486714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.486717:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575110.486720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495685120:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575110.486735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495685248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575110.486752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575110.486760:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff93a25064f100 x1875453495685248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.486767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495685248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.486772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.486774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575110.486781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495685248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575111.446320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575111.446380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575112.470322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575112.470369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575113.494300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575113.494345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575114.518323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575114.518381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575115.542283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575115.542359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575115.606319:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.606327:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575115.606347:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.606356:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3332 00000100:00080000:3.0:1788575115.606365:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.606372:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575115.606403:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495685376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.606423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.606426:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575115.606427:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.606430:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3332 00000100:00080000:3.0:1788575115.606432:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.606434:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575115.606441:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575115.606449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495685504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.606455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.606457:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575115.606458:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.606461:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3332 00000100:00080000:3.0:1788575115.606462:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.606464:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575115.606467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575115.606473:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495685376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575115.606477:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495685632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.606500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.606503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575115.606505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.606510:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3332 00000100:00080000:3.0:1788575115.606514:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.606518:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575115.606522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575115.606533:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575115.606534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495685760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575115.606593:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff93a25064d180 x1875453495685376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.606615:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495685376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.606626:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.606630:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575115.606634:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495685376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575115.606641:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495685504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575115.606662:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575115.606674:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff93a25064c000 x1875453495685504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.606686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495685504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.606695:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.606698:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575115.606701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495685504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575115.606726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495685632:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575115.606752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575115.606774:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff93a25064f480 x1875453495685632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.606785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495685632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.606794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.606798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575115.606802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495685632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575115.606808:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495685760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575115.606827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575115.606837:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff93a25064fb80 x1875453495685760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.606849:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495685760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.606857:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.606860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575115.606863:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495685760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575116.566275:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575116.566392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575117.590330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575117.590375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575118.614283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575118.614360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575119.638304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575119.638351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575120.662303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575120.662364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575120.726351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575120.726373:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575120.726376:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575120.726384:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3337 00000100:00080000:0.0:1788575120.726391:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575120.726396:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575120.726421:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495685888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575120.726441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575120.726443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575120.726445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575120.726447:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3337 00000100:00080000:0.0:1788575120.726449:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575120.726451:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575120.726458:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575120.726466:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495686016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575120.726472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575120.726474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575120.726476:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575120.726478:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3337 00000100:00080000:0.0:1788575120.726480:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575120.726483:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575120.726485:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495685888:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788575120.726487:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575120.726496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495686144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575120.726509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575120.726511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575120.726512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575120.726515:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3337 00000100:00080000:0.0:1788575120.726517:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575120.726519:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575120.726523:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575120.726530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495686272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575120.726541:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575120.726577:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff93a2460d0700 x1875453495685888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.726590:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495685888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.726602:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.726605:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575120.726608:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495685888:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575120.726612:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575120.726629:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575120.726636:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff93a2460d0a80 x1875453495686016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.726643:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495686016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.726648:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.726649:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575120.726651:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575120.726667:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686144:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575120.726677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575120.726683:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff93a2460d0000 x1875453495686144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.726689:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495686144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.726694:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.726695:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575120.726697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686144:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575120.726705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495686272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575120.726717:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575120.726722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff93a2460d3b80 x1875453495686272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.726729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495686272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.726733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.726734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575120.726736:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495686272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575121.686329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575121.686387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575122.710309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575122.710383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575123.734330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575123.734383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575124.758327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575124.758406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575125.783105:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575125.783304:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575125.846324:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575125.846331:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575125.846335:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575125.846344:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3342 00000100:00080000:0.0:1788575125.846351:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575125.846357:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575125.846387:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495686400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575125.846404:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575125.846407:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575125.846419:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575125.846424:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3342 00000100:00080000:0.0:1788575125.846430:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575125.846434:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575125.846444:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575125.846453:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495686400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575125.846460:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495686528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575125.846473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575125.846476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575125.846477:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575125.846481:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3342 00000100:00080000:0.0:1788575125.846484:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575125.846488:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575125.846492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575125.846502:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495686656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575125.846510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575125.846513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575125.846514:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575125.846518:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3342 00000100:00080000:0.0:1788575125.846520:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575125.846523:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575125.846526:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575125.846541:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495686784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575125.846556:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575125.846584:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575125.846605:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495686784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575126.807783:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575126.807838:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575127.830556:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575127.830792:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575128.854697:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575128.854862:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575128.918481:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788575128.918516:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575128.918528:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575128.918566:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575128.918575:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575128.918583:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575128.918665:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575128] req@ffff93a2460d0000 x1875453495686528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575128.918692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495686528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575128.918703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575128.918708:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575128.918729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575128.918738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575128] req@ffff93a2460d0a80 x1875453495686656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575128.918751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495686656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575128.918759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575128.918762:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575128.918765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495686656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575128.918772:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575128] req@ffff93a2460d0700 x1875453495686784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575128.918783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495686784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575128.918790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575128.918794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575128.918797:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495686784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575128.918804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575128] req@ffff93a2460d3b80 x1875453495686400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575128.918816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495686400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575128.918823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575128.918826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575128.918836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495686400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575129.878447:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575129.879218:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575130.902315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575130.902384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575130.966333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575130.966342:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575130.966346:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575130.966372:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3347 00000100:00080000:3.0:1788575130.966381:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575130.966388:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575130.966417:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495686912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575130.966441:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575130.966444:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575130.966446:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575130.966451:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3347 00000100:00080000:3.0:1788575130.966454:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575130.966458:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575130.966480:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575130.966492:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495686912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575130.966496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495687040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575130.966508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575130.966512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575130.966513:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575130.966518:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3347 00000100:00080000:3.0:1788575130.966527:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575130.966532:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575130.966537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575130.966547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495687168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575130.966556:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575130.966563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575130.966567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575130.966570:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575130.966574:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3347 00000100:00080000:3.0:1788575130.966578:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575130.966582:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575130.966587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575130.966598:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495687296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575130.966613:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff93a25064e680 x1875453495686912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.966633:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495686912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.966643:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.966647:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575130.966651:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495686912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575130.966678:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575130.966700:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575130.966717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff93a25064ea00 x1875453495687040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.966728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495687040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.966735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.966739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575130.966742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575130.966748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687168:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575130.966767:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575130.966800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff93a25064d880 x1875453495687168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.966812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495687168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.966819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.966823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575130.966826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687168:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575130.966832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495687296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575130.966851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575130.966861:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff93a25064dc00 x1875453495687296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.966886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495687296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.966894:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.966897:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575130.966900:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495687296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575131.926352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575131.926402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575132.950309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575132.950400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575133.974368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575133.974432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575134.998370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575134.999476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575136.022285:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575136.022404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575136.086336:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575136.086346:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575136.086350:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575136.086361:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3352 00000100:00080000:1.0:1788575136.086383:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575136.086395:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575136.086454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495687424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575136.086492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575136.086496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575136.086499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575136.086503:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3352 00000100:00080000:1.0:1788575136.086508:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575136.086512:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575136.086523:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575136.086537:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495687552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575136.086541:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495687424:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575136.086556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575136.086560:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575136.086562:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575136.086567:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3352 00000100:00080000:1.0:1788575136.086571:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575136.086575:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575136.086581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575136.086593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495687680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575136.086607:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575136.086614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575136.086621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575136.086624:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575136.086634:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3352 00000100:00080000:1.0:1788575136.086639:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575136.086643:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575136.086649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575136.086650:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff93a258da0380 x1875453495687424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575136.086664:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495687808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575136.086672:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495687424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575136.086683:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575136.086686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575136.086689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495687424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575136.086707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575136.086724:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575136.086731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff93a258da0000 x1875453495687552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575136.086738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495687552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575136.086749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575136.086751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575136.086754:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575136.086757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687680:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575136.086771:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575136.086777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff93a258da2300 x1875453495687680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575136.086783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495687680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575136.086788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575136.086790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575136.086791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495687680:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575136.086795:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495687808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575136.086804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575136.086809:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff93a258da1180 x1875453495687808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575136.086816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495687808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575136.086820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575136.086821:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575136.086828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495687808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575137.046304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575137.046357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575138.070319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575138.070409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575139.094370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575139.094435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575140.118305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575140.118355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575141.142307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575141.142372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575141.206371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.206382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575141.206387:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.206397:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3357 00000100:00080000:3.0:1788575141.206407:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.206415:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575141.206454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495687936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.206480:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.206484:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575141.206486:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.206490:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3357 00000100:00080000:3.0:1788575141.206494:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.206499:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575141.206508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575141.206522:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495687936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575141.206524:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495688064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.206539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.206543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575141.206545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.206563:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3357 00000100:00080000:3.0:1788575141.206569:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.206573:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575141.206579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575141.206583:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575141.206595:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495688192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.206607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.206611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575141.206613:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.206617:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3357 00000100:00080000:3.0:1788575141.206622:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575141.206624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff93a25064ed80 x1875453495687936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575141.206629:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575141.206637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575141.206649:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495687936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575141.206653:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495688320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575141.206673:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575141.206680:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575141.206685:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495687936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575141.206709:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575141.206731:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575141.206743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff93a25064dc00 x1875453495688064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575141.206754:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495688064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575141.206762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575141.206766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575141.206768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575141.206774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688192:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575141.206794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575141.206812:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff93a25064d880 x1875453495688192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575141.206823:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495688192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575141.206831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575141.206834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575141.206837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575141.206842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495688320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575141.206857:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575141.206865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff93a25064ea00 x1875453495688320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575141.206875:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495688320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575141.206882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575141.206885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575141.206887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495688320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575142.166313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575142.166407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575143.190358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575143.190410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575144.214330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575144.214380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575145.238313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575145.238368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575146.262312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575146.262361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575146.326311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.326321:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575146.326326:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.326334:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3362 00000100:00080000:3.0:1788575146.326343:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.326363:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575146.326390:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495688448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.326412:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.326414:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575146.326415:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.326418:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3362 00000100:00080000:3.0:1788575146.326420:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.326422:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575146.326430:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575146.326437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495688576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.326444:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.326446:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575146.326448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575146.326452:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495688448:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575146.326452:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3362 00000100:00080000:3.0:1788575146.326454:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.326460:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575146.326464:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575146.326471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495688704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.326477:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.326480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575146.326485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.326488:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3362 00000100:00080000:3.0:1788575146.326490:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.326493:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575146.326497:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575146.326503:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495688832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575146.326513:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575146.326549:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff93a25064e680 x1875453495688448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575146.326561:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495688448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575146.326574:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575146.326577:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575146.326581:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495688448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575146.326598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575146.326618:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575146.326630:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff93a25064fb80 x1875453495688576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575146.326641:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495688576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575146.326649:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575146.326652:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575146.326656:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575146.326662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688704:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575146.326681:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575146.326691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff93a25064f480 x1875453495688704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575146.326702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495688704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575146.326710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575146.326713:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575146.326716:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495688704:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575146.326729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495688832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575146.326750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575146.326760:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff93a25064c000 x1875453495688832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575146.326771:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495688832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575146.326776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575146.326778:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575146.326780:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495688832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575147.286304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575147.286401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575148.310295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575148.310351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575149.334289:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575149.334361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575150.358296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575150.358345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575151.382324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575151.382378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575151.446341:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575151.446353:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575151.446357:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575151.446368:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3367 00000100:00080000:0.0:1788575151.446378:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575151.446386:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575151.446424:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495688960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575151.446455:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575151.446459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575151.446460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575151.446464:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3367 00000100:00080000:0.0:1788575151.446467:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575151.446486:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575151.446496:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575151.446502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495688960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575151.446511:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495689088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575151.446521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575151.446523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575151.446525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575151.446528:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3367 00000100:00080000:0.0:1788575151.446530:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575151.446532:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575151.446537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575151.446544:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495689216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575151.446551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575151.446554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575151.446555:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575151.446558:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3367 00000100:00080000:0.0:1788575151.446573:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575151.446576:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575151.446581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575151.446586:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575151.446597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495689344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575151.446626:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff93a2460d3480 x1875453495688960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.446640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495688960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.446647:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.446650:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575151.446653:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495688960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575151.446671:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575151.446686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575151.446694:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff93a2460d1500 x1875453495689088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.446702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495689088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.446708:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.446715:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575151.446718:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575151.446722:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689216:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575151.446738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575151.446744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff93a2460d2680 x1875453495689216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.446752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495689216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.446757:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.446759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575151.446761:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575151.446764:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495689344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575151.446774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575151.446780:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff93a2460d0380 x1875453495689344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.446787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495689344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.446792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.446793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575151.446795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495689344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575152.406288:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575152.406357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575153.430284:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575153.430317:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575154.454291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575154.454376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575155.478334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575155.478396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575156.502315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575156.502371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575156.566361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.566387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575156.566390:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.566399:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3373 00000100:00080000:0.0:1788575156.566407:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.566414:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575156.566444:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495689472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575156.566471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.566475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575156.566477:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.566482:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3373 00000100:00080000:0.0:1788575156.566486:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.566491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575156.566501:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575156.566516:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495689600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575156.566516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495689472:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575156.566524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.566530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575156.566532:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.566535:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3373 00000100:00080000:0.0:1788575156.566537:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.566540:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575156.566544:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575156.566552:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495689728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575156.566565:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.566567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575156.566569:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.566572:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3373 00000100:00080000:0.0:1788575156.566575:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.566577:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575156.566581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575156.566588:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575156.566590:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495689856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575156.566632:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff93a2460d2300 x1875453495689472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.566647:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495689472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.566656:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.566659:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575156.566662:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495689472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575156.566682:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575156.566699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575156.566708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff93a2460d1f80 x1875453495689600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.566716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495689600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.566722:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.566725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575156.566727:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575156.566731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689728:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575156.566742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575156.566748:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff93a2460d2a00 x1875453495689728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.566767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495689728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.566773:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.566775:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575156.566778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495689728:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575156.566781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495689856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575156.566794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575156.566800:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff93a2460d1180 x1875453495689856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.566808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495689856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.566814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.566815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575156.566817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495689856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575157.526295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575157.526365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575158.550379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575158.550449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575159.574310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575159.574382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575160.598358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575160.598423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575161.622309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575161.622364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575161.686416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.686429:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575161.686433:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.686445:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3378 00000100:00080000:1.0:1788575161.686457:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.686464:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575161.686509:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495689984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575161.686564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.686567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575161.686568:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.686572:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3378 00000100:00080000:1.0:1788575161.686575:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.686579:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575161.686589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575161.686600:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495690112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575161.686610:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495689984:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575161.686610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.686616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575161.686619:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.686625:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3378 00000100:00080000:1.0:1788575161.686627:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.686629:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575161.686632:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575161.686640:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495690240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575161.686647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.686650:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575161.686651:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.686653:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3378 00000100:00080000:1.0:1788575161.686662:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.686665:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575161.686669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575161.686677:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495690368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575161.686706:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575161.686769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff93a258da3480 x1875453495689984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.686793:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495689984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.686806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.686810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575161.686815:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495689984:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575161.686845:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575161.686872:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575161.686886:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff93a258da2a00 x1875453495690112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.686907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495690112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.686913:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.686916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575161.686918:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575161.686923:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690240:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575161.686962:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575161.686971:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff93a258da3100 x1875453495690240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.686980:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495690240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.686986:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.686988:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575161.686990:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690240:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575161.686994:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495690368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575161.687009:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575161.687015:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff93a258da1880 x1875453495690368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.687023:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495690368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.687040:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.687043:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575161.687045:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495690368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575162.646323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575162.646389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575163.670332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575163.670387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575164.694327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575164.694380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575165.718315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575165.718358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575166.742340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575166.742395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575166.807759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575166.807767:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575166.807771:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575166.807781:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3383 00000100:00080000:0.0:1788575166.807792:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575166.807800:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575166.807839:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495690496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575166.807871:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575166.807875:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575166.807878:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575166.807882:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3383 00000100:00080000:0.0:1788575166.807886:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575166.807891:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575166.807902:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575166.807920:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495690496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575166.807921:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495690624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575166.807940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575166.807943:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575166.807961:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575166.807967:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3383 00000100:00080000:0.0:1788575166.807982:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575166.807986:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575166.807991:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575166.807995:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575166.808005:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495690752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575166.808014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575166.808017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575166.808018:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575166.808021:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3383 00000100:00080000:0.0:1788575166.808025:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575166.808029:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575166.808035:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575166.808043:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff93a2460d1180 x1875453495690496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575166.808050:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495690880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575166.808063:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495690496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.808073:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.808077:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575166.808080:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495690496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575166.808101:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575166.808140:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575166.808154:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff93a2460d2a00 x1875453495690624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.808167:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495690624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.808176:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.808180:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575166.808196:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575166.808203:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690752:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575166.808243:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575166.808254:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff93a2460d1f80 x1875453495690752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.808262:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495690752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.808268:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.808270:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575166.808273:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495690752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575166.808276:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495690880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575166.808290:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575166.808296:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff93a2460d2300 x1875453495690880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.808303:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495690880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.808308:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.808310:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575166.808312:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495690880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575167.766307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575167.766408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575168.790319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575168.790385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575169.814374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575169.814470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575170.838371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575170.838411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575171.862345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575171.862413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575171.926396:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575171.926411:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575171.926417:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575171.926430:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3388 00000100:00080000:0.0:1788575171.926445:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575171.926454:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575171.926503:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495691008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575171.926555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575171.926562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575171.926564:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575171.926568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3388 00000100:00080000:0.0:1788575171.926571:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575171.926575:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575171.926585:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575171.926597:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495691136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575171.926605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575171.926608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575171.926610:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575171.926615:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3388 00000100:00100000:1.0:1788575171.926615:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495691008:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575171.926619:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575171.926626:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575171.926630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575171.926642:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495691264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575171.926653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575171.926657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575171.926659:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575171.926664:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3388 00000100:00080000:0.0:1788575171.926695:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575171.926707:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575171.926713:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575171.926722:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575171.926738:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495691392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575171.926778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff93a2460d0380 x1875453495691008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575171.926801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495691008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575171.926819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575171.926823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575171.926828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495691008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575171.926860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575171.926886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575171.926898:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff93a2460d2680 x1875453495691136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575171.926907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495691136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575171.926913:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575171.926915:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575171.926917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575171.926921:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691264:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575171.926934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575171.926941:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff93a2460d1500 x1875453495691264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575171.926949:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495691264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575171.926964:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575171.926966:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575171.926969:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691264:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575171.926973:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495691392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575171.926987:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575171.926994:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff93a2460d3480 x1875453495691392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575171.927001:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495691392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575171.927006:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575171.927008:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575171.927010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495691392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575172.886334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575172.886376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575173.910326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575173.910393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575174.934344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575174.934404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575175.958349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575175.958419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575176.982365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575176.982426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575177.046319:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575177.046329:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575177.046333:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575177.046343:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3393 00000100:00080000:1.0:1788575177.046354:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575177.046360:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575177.046400:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495691520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575177.046428:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575177.046446:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575177.046448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575177.046452:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3393 00000100:00080000:1.0:1788575177.046455:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575177.046459:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575177.046471:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575177.046481:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495691520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575177.046484:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495691648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575177.046507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575177.046516:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575177.046517:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575177.046521:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3393 00000100:00080000:1.0:1788575177.046523:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575177.046526:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575177.046530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575177.046539:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495691776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575177.046545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575177.046548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575177.046549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575177.046552:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3393 00000100:00080000:1.0:1788575177.046563:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575177.046566:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788575177.046570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788575177.046582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575177.046594:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495691904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575177.046626:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff93a258da3800 x1875453495691520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.046644:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495691520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.046652:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.046655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575177.046657:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495691520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575177.046676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575177.046698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575177.046706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff93a258da1c00 x1875453495691648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.046713:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495691648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.046718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.046721:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575177.046734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575177.046738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691776:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575177.046752:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575177.046758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff93a258da0700 x1875453495691776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.046764:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495691776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.046768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.046770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575177.046772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495691776:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575177.046775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495691904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575177.046784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575177.046789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff93a258da1f80 x1875453495691904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.046795:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495691904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.046799:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.046800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575177.046802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495691904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575178.006369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575178.006424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575179.030308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575179.030345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788575180.054345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575180.054346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575181.078301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575181.078333:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575182.102331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575182.102388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575182.166346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575182.166364:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575182.166385:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575182.166395:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3398 00000100:00080000:0.0:1788575182.166405:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575182.166412:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575182.166447:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495692032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575182.166475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575182.166478:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575182.166480:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575182.166484:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3398 00000100:00080000:0.0:1788575182.166487:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575182.166491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575182.166500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575182.166513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495692160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575182.166518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495692032:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575182.166530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575182.166534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575182.166536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575182.166540:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3398 00000100:00080000:0.0:1788575182.166544:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575182.166547:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575182.166552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575182.166564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495692288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575182.166586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575182.166591:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575182.166592:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575182.166597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575182.166602:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3398 00000100:00080000:0.0:1788575182.166607:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575182.166612:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575182.166618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575182.166631:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495692416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575182.166636:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff93a2460d3b80 x1875453495692032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575182.166659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495692032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575182.166669:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575182.166673:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575182.166676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495692032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575182.166696:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575182.166728:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575182.166740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff93a2460d0700 x1875453495692160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575182.166759:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495692160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575182.166767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575182.166770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575182.166773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575182.166778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692288:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575182.166804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575182.166815:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff93a2460d0a80 x1875453495692288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575182.166826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495692288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575182.166834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575182.166837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575182.166840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575182.166846:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495692416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575182.166870:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575182.166878:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff93a2460d0000 x1875453495692416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575182.166885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495692416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575182.166890:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575182.166892:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575182.166893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495692416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575183.126330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575183.126388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575184.150321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575184.150371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575185.174345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575185.174382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575186.198322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575186.198375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575187.222357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575187.222384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575187.286325:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575187.286343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575187.286347:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575187.286357:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3403 00000100:00080000:0.0:1788575187.286366:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575187.286372:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575187.286416:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495692544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575187.286436:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575187.286438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575187.286440:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575187.286443:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3403 00000100:00080000:0.0:1788575187.286445:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575187.286448:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575187.286479:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575187.286481:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495692544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575187.286490:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495692672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575187.286498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575187.286501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575187.286502:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575187.286504:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3403 00000100:00080000:0.0:1788575187.286507:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575187.286510:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575187.286515:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575187.286525:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495692800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575187.286534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575187.286538:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575187.286542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575187.286545:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575187.286550:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3403 00000100:00080000:0.0:1788575187.286554:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575187.286559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575187.286564:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575187.286574:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff93a2460d0000 x1875453495692544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575187.286577:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495692928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575187.286605:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495692544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575187.286613:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575187.286615:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575187.286618:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495692544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575187.286635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575187.286650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575187.286657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff93a2460d0a80 x1875453495692672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575187.286664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495692672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575187.286669:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575187.286671:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575187.286673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575187.286676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692800:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575187.286685:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575187.286690:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff93a2460d0700 x1875453495692800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575187.286696:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495692800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575187.286708:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575187.286710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575187.286712:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495692800:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575187.286715:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495692928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575187.286726:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575187.286732:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff93a2460d3b80 x1875453495692928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575187.286738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495692928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575187.286747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575187.286748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575187.286750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495692928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575188.246320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575188.246365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575189.270365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575189.270428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575190.294330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575190.294354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788575191.318478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575191.318479:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:1.0:1788575192.342575:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575192.342731:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575192.406352:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575192.406369:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575192.406373:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575192.406383:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3408 00000100:00080000:1.0:1788575192.406393:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575192.406401:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575192.406432:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495693056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575192.406454:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575192.406473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575192.406475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575192.406480:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495693056:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575192.406483:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3408 00000100:00080000:1.0:1788575192.406488:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575192.406493:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575192.406506:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575192.406522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495693184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575192.406533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575192.406537:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575192.406539:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575192.406543:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3408 00000100:00080000:1.0:1788575192.406547:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575192.406551:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575192.406556:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575192.406568:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495693312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575192.406572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693184:192.168.202.102@tcp:38:kworker.0 00000100:00080000:1.0:1788575192.406586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575192.406591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575192.406593:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575192.406597:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3408 00000100:00100000:0.0:1788575192.406600:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693312:192.168.202.102@tcp:8:kworker.0 00000100:00080000:1.0:1788575192.406603:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575192.406610:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575192.406622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575192.406635:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495693440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575192.406676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495693440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575193.366388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575193.366442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575194.390872:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575194.390989:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575195.414623:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575195.414985:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:1.0:1788575195.478526:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788575195.478573:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575195.478589:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575195.478625:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575195.478635:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575195.478643:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788575195.478670:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff93a258da1f80 x1875453495693440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.478694:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495693440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.478705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.478710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575195.478714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495693440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575195.478722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff93a258da0a80 x1875453495693312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.478733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495693312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.478740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.478744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575195.478747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693312:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575195.478754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff93a258da2680 x1875453495693184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.478787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495693184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.478796:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.478800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575195.478803:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575195.478810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff93a258da0e00 x1875453495693056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.478822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495693056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.478830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.478832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575195.478835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495693056:192.168.202.102@tcp:250:kworker.0 00010000:00010000:0.0:1788575196.438355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575196.438464:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575197.462322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575197.462370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575197.526327:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.526335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575197.526338:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.526346:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3413 00000100:00080000:1.0:1788575197.526354:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.526361:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575197.526393:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495693568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.526419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.526422:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575197.526424:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.526428:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3413 00000100:00080000:1.0:1788575197.526432:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.526436:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575197.526445:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575197.526460:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495693568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575197.526482:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495693696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.526493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.526496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575197.526498:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.526502:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3413 00000100:00080000:1.0:1788575197.526520:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.526524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788575197.526530:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788575197.526531:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575197.526545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495693824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.526554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.526557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575197.526559:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.526563:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3413 00000100:00080000:1.0:1788575197.526566:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.526570:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575197.526574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575197.526584:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff93a258da0700 x1875453495693568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575197.526586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495693952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575197.526624:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495693568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.526637:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.526647:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575197.526652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495693568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575197.526676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575197.526699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575197.526712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff93a258da1c00 x1875453495693696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.526738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495693696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.526747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.526750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575197.526753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575197.526760:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693824:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575197.526813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575197.526826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff93a258da3800 x1875453495693824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.526838:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495693824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.526845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.526849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575197.526852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495693824:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575197.526858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495693952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575197.526885:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575197.526896:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff93a258da1880 x1875453495693952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.526919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495693952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.526926:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.526929:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575197.526932:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495693952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575198.486334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575198.486353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575199.510372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575199.510434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575200.534279:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575200.534400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575201.558322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575201.558367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575202.582309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575202.582371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575202.646298:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575202.646306:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575202.646309:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575202.646332:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3419 00000100:00080000:1.0:1788575202.646342:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575202.646348:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575202.646379:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495694080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575202.646405:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575202.646408:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575202.646410:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575202.646419:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3419 00000100:00080000:1.0:1788575202.646424:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575202.646428:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575202.646438:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575202.646441:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495694080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575202.646453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495694208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575202.646464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575202.646468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575202.646470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575202.646475:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3419 00000100:00080000:1.0:1788575202.646486:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575202.646491:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575202.646507:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575202.646514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575202.646528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495694336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575202.646543:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff93a258da3100 x1875453495694080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575202.646563:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575202.646567:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495694080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575202.646567:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575202.646575:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575202.646576:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575202.646581:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3419 00000100:00080000:0.0:1788575202.646583:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00080000:1.0:1788575202.646586:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575202.646590:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495694080:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575202.646592:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575202.646599:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575202.646609:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495694464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575202.646620:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575202.646657:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575202.646667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff93a258da2a00 x1875453495694208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.646675:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495694208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.646680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.646682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575202.646684:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575202.646688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694336:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575202.646699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575202.646705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff93a258da3480 x1875453495694336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.646711:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495694336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.646716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.646717:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575202.646719:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694336:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575202.646729:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495694464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575202.646738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575202.646744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff93a258da1180 x1875453495694464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.646761:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495694464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.646771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.646774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575202.646775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495694464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575203.606349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575203.606426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575204.630299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575204.630373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575205.654299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575205.654361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575206.678376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575206.678468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575207.702337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575207.702417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575207.766465:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.766478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575207.766483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.766495:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3424 00000100:00080000:2.0:1788575207.766506:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.766513:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575207.766556:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495694592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.766583:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.766588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575207.766594:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.766599:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3424 00000100:00080000:2.0:1788575207.766603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.766607:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575207.766620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575207.766642:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495694592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575207.766642:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495694720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.766653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.766657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575207.766659:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.766663:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3424 00000100:00080000:2.0:1788575207.766666:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.766671:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575207.766676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575207.766704:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495694848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.766715:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.766718:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1788575207.766719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575207.766720:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.766724:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3424 00000100:00080000:2.0:1788575207.766727:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.766731:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575207.766736:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575207.766754:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495694976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575207.766791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff93a24940ad80 x1875453495694592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.766812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495694592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.766825:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.766829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575207.766834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495694592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575207.766865:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575207.766894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575207.766919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff93a249409180 x1875453495694720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.766934:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495694720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.766958:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.766961:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575207.766964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575207.766968:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694848:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575207.766988:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575207.766999:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff93a249408a80 x1875453495694848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.767011:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495694848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.767020:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.767024:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575207.767028:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495694848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575207.767035:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495694976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575207.767067:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575207.767076:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff93a249408700 x1875453495694976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.767084:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495694976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.767090:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.767092:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575207.767100:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495694976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575208.726326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575208.726376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575209.750370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575209.750447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575210.774340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575210.774416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575211.798361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575211.798448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575212.822288:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575212.822353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575212.886311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.886318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575212.886322:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.886329:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3429 00000100:00080000:0.0:1788575212.886337:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.886342:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575212.886369:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495695104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575212.886389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.886391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575212.886392:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.886395:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3429 00000100:00080000:0.0:1788575212.886397:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.886399:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575212.886405:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575212.886413:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495695232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575212.886419:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.886422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575212.886423:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.886426:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3429 00000100:00080000:0.0:1788575212.886429:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575212.886429:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495695104:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575212.886446:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575212.886456:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575212.886465:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495695360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575212.886471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.886474:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575212.886475:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.886479:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3429 00000800:00000100:1.0:1788575212.886480:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575212.886484:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.886487:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575212.886490:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575212.886497:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495695488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575212.886518:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff93a2460d3480 x1875453495695104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575212.886532:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495695104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575212.886540:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575212.886544:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575212.886548:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495695104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575212.886569:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575212.886590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575212.886600:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff93a2460d1500 x1875453495695232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575212.886614:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495695232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575212.886619:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575212.886623:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575212.886626:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575212.886631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695360:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575212.886652:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575212.886663:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff93a2460d2680 x1875453495695360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575212.886673:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495695360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575212.886679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575212.886682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575212.886686:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695360:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575212.886691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495695488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575212.886709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575212.886720:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff93a2460d0380 x1875453495695488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575212.886728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495695488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575212.886741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575212.886743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575212.886746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495695488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575213.846305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575213.846350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575214.870320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575214.870377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575215.894315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575215.894349:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575216.918314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575216.918373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575217.942308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575217.942360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575218.006344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575218.006353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575218.006357:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575218.006366:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3434 00000100:00080000:2.0:1788575218.006374:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575218.006395:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575218.006423:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495695616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575218.006446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575218.006450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575218.006452:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575218.006457:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3434 00000100:00080000:2.0:1788575218.006461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575218.006466:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575218.006476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575218.006488:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495695616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575218.006493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495695744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575218.006508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575218.006511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575218.006512:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575218.006515:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3434 00000100:00080000:2.0:1788575218.006519:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575218.006522:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575218.006526:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575218.006535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495695872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575218.006542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575218.006546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1788575218.006547:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575218.006553:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575218.006558:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3434 00000100:00080000:2.0:1788575218.006562:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575218.006566:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575218.006572:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575218.006587:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff93a249409880 x1875453495695616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575218.006588:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495696000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575218.006617:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495695616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575218.006626:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575218.006630:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575218.006633:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495695616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575218.006653:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575218.006669:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575218.006677:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff93a249408000 x1875453495695744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575218.006685:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495695744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575218.006691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575218.006693:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575218.006695:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575218.006707:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695872:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575218.006721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575218.006728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff93a24940aa00 x1875453495695872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575218.006737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495695872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575218.006743:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575218.006745:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575218.006747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495695872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575218.006751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495696000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575218.006761:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575218.006766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff93a249409500 x1875453495696000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575218.006773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495696000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575218.006779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575218.006781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575218.006783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495696000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575218.966335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575218.966386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575219.990296:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575219.990356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575221.014373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575221.014456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575222.038291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575222.038392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575223.062301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575223.062390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575223.126388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.126399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575223.126403:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.126415:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3439 00000100:00080000:3.0:1788575223.126425:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.126433:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575223.126468:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495696128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.126499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.126503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575223.126505:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.126527:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3439 00000100:00080000:3.0:1788575223.126532:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.126537:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575223.126546:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495696128:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788575223.126549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575223.126568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495696256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.126580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.126583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575223.126585:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.126589:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3439 00000100:00080000:3.0:1788575223.126592:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.126596:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575223.126602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575223.126609:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575223.126616:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495696384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.126628:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.126632:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575223.126633:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.126638:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3439 00000100:00080000:3.0:1788575223.126642:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.126646:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575223.126668:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575223.126681:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495696512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575223.126698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff93a25064d180 x1875453495696128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575223.126730:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495696128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575223.126743:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575223.126748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575223.126752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495696128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575223.126776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575223.126802:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575223.126817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff93a25064f100 x1875453495696256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575223.126832:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495696256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575223.126842:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575223.126846:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575223.126850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575223.126857:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696384:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575223.126880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575223.126893:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff93a25064ce00 x1875453495696384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575223.126907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495696384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575223.126930:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575223.126934:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575223.126939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696384:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575223.126946:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495696512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575223.126971:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575223.126983:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff93a25064e300 x1875453495696512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575223.126996:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495696512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575223.127005:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575223.127024:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575223.127028:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495696512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575224.086315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575224.086416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575225.110328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575225.110410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575226.134327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575226.134379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575227.158348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575227.158426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575228.182340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575228.182452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575228.246410:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.246424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575228.246430:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.246454:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3444 00000100:00080000:2.0:1788575228.246466:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.246475:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575228.246529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495696640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575228.246561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.246583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575228.246585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.246589:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3444 00000100:00080000:2.0:1788575228.246595:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.246600:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575228.246613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575228.246624:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495696640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575228.246642:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495696768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575228.246657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.246661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575228.246663:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.246669:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3444 00000100:00080000:2.0:1788575228.246674:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.246679:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575228.246685:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575228.246699:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495696896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575228.246712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575228.246713:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.246719:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575228.246721:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.246727:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3444 00000100:00080000:2.0:1788575228.246733:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.246738:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575228.246751:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575228.246765:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495697024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575228.246774:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff93a24940b480 x1875453495696640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575228.246797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495696640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575228.246809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575228.246814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575228.246819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495696640:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575228.246858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575228.246883:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575228.246896:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff93a24940bb80 x1875453495696768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575228.246908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495696768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575228.246916:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575228.246920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575228.246923:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575228.246929:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696896:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575228.246947:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575228.246957:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff93a24940b100 x1875453495696896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575228.246968:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495696896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575228.246976:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575228.246979:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575228.246982:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495696896:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575228.246989:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495697024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575228.247008:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575228.247019:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff93a249409500 x1875453495697024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575228.247038:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495697024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575228.247046:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575228.247049:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575228.247052:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495697024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575229.206357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575229.206438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575230.230330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575230.230420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575231.254334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575231.254379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575232.278320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575232.278417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575233.302310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575233.302415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575233.366342:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575233.366356:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575233.366361:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575233.366375:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3449 00000100:00080000:2.0:1788575233.366388:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575233.366398:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575233.366441:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495697152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575233.366472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575233.366476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575233.366478:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575233.366482:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3449 00000100:00080000:2.0:1788575233.366486:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575233.366491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575233.366504:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575233.366519:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495697280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575233.366549:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495697152:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575233.366557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575233.366562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575233.366564:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575233.366571:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3449 00000100:00080000:2.0:1788575233.366576:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575233.366582:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575233.366588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575233.366603:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495697408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575233.366614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575233.366618:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575233.366620:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575233.366624:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3449 00000100:00080000:2.0:1788575233.366628:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575233.366632:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575233.366638:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575233.366644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575233.366658:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495697536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575233.366688:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff93a24940aa00 x1875453495697152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575233.366704:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495697152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575233.366713:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575233.366716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575233.366727:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495697152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575233.366747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575233.366766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575233.366775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff93a249408000 x1875453495697280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575233.366784:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495697280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575233.366814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575233.366817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575233.366820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575233.366824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697408:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575233.366842:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575233.366850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff93a249409880 x1875453495697408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575233.366857:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495697408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575233.366863:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575233.366865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575233.366867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575233.366871:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495697536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575233.366882:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575233.366888:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff93a249408700 x1875453495697536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575233.366895:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495697536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575233.366900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575233.366908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575233.366916:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495697536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575234.326312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575234.326403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575235.350348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575235.350412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575236.374332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575236.374382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575237.398311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575237.398413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575238.422310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575238.422360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575238.486326:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.486334:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575238.486339:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.486347:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3454 00000100:00080000:3.0:1788575238.486355:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575238.486360:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575238.486391:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495697664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.486416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.486420:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575238.486422:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.486428:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3454 00000100:00080000:3.0:1788575238.486447:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575238.486450:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495697664:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575238.486456:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575238.486469:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575238.486486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495697792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.486495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.486498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575238.486499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.486502:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3454 00000100:00080000:3.0:1788575238.486506:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575238.486508:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575238.486513:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575238.486518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575238.486531:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495697920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.486543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575238.486545:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff93a25064d500 x1875453495697664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575238.486550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575238.486556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.486563:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3454 00000100:00080000:3.0:1788575238.486568:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575238.486569:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495697664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575238.486577:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575238.486587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788575238.486592:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575238.486600:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:3.0:1788575238.486605:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495698048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575238.486610:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495697664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575238.486645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575238.486671:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575238.486680:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff93a25064c700 x1875453495697792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575238.486690:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495697792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575238.486699:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575238.486702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575238.486706:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575238.486712:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697920:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575238.486734:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575238.486754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff93a25064ca80 x1875453495697920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575238.486762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495697920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575238.486770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575238.486773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575238.486777:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495697920:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575238.486783:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495698048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575238.486804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575238.486816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff93a25064c380 x1875453495698048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575238.486827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495698048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575238.486832:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575238.486834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575238.486836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495698048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575239.446304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575239.446356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575240.470326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575240.470377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575241.494384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575241.494428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575242.518368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575242.518422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575243.542330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575243.542385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575243.606344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575243.606356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575243.606360:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575243.606370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3460 00000100:00080000:0.0:1788575243.606381:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575243.606388:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575243.606426:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495698176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575243.606454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575243.606458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575243.606473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575243.606479:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3460 00000100:00080000:0.0:1788575243.606484:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575243.606498:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575243.606502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495698176:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788575243.606511:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575243.606531:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495698304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575243.606544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575243.606549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575243.606551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575243.606557:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3460 00000800:00000100:1.0:1788575243.606559:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575243.606567:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575243.606574:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575243.606581:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575243.606596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495698432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575243.606601:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff93a2460d2300 x1875453495698176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.606616:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575243.606622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575243.606625:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575243.606628:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495698176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.606633:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3460 00000100:00080000:0.0:1788575243.606639:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575243.606649:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:1.0:1788575243.606650:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788575243.606658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788575243.606658:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575243.606662:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495698176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575243.606683:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495698560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575243.606689:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575243.606739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575243.606758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff93a2460d1f80 x1875453495698304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575243.606768:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495698304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575243.606774:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575243.606776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575243.606779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575243.606789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698432:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575243.606843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575243.606857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff93a2460d2a00 x1875453495698432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575243.606871:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495698432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575243.606880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575243.606884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575243.606888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698432:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575243.606908:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495698560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575243.606925:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575243.606933:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff93a2460d1180 x1875453495698560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575243.606941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495698560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575243.606946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575243.606949:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575243.606952:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495698560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575244.566368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575244.566458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575245.590334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575245.590391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575246.614345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575246.614405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575247.638318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575247.638393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575248.662363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575248.662426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575248.726342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575248.726353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575248.726358:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575248.726370:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3465 00000100:00080000:0.0:1788575248.726382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575248.726390:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575248.726429:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495698688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575248.726451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575248.726456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575248.726458:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575248.726462:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3465 00000100:00080000:0.0:1788575248.726467:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575248.726471:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575248.726481:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575248.726496:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495698816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575248.726506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495698688:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575248.726510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575248.726515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575248.726517:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575248.726523:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3465 00000100:00080000:0.0:1788575248.726527:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575248.726550:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575248.726557:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575248.726561:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575248.726575:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495698944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575248.726588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575248.726606:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575248.726609:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575248.726614:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff93a2460d1180 x1875453495698688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575248.726617:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3465 00000100:00080000:0.0:1788575248.726626:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575248.726632:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575248.726637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495698688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575248.726640:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788575248.726651:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575248.726654:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575248.726661:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495699072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575248.726662:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495698688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575248.726689:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575248.726708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575248.726717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff93a2460d2a00 x1875453495698816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575248.726732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495698816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575248.726738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575248.726740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575248.726750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575248.726754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698944:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575248.726770:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575248.726777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff93a2460d1f80 x1875453495698944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575248.726785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495698944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575248.726790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575248.726792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575248.726794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495698944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575248.726798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495699072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575248.726808:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575248.726813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff93a2460d2300 x1875453495699072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575248.726820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495699072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575248.726835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575248.726837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575248.726839:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495699072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575249.686304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575249.686354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575250.710327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575250.710382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575251.734307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575251.734359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575252.758309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575252.758355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575253.782314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575253.782372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575253.846356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575253.846364:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575253.846368:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575253.846376:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3470 00000100:00080000:3.0:1788575253.846405:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575253.846412:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575253.846450:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495699200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575253.846470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575253.846473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575253.846474:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575253.846477:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3470 00000100:00080000:3.0:1788575253.846495:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575253.846499:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575253.846508:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575253.846509:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495699200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575253.846529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495699328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575253.846539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575253.846542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575253.846543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575253.846547:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3470 00000100:00080000:3.0:1788575253.846550:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575253.846552:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575253.846555:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575253.846563:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495699456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575253.846571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575253.846574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575253.846575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575253.846579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575253.846583:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3470 00000100:00080000:3.0:1788575253.846586:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575253.846588:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575253.846592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575253.846604:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495699584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575253.846621:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff93a25064df80 x1875453495699200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575253.846637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495699200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575253.846646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575253.846649:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575253.846653:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495699200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575253.846680:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575253.846714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575253.846724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff93a25064c380 x1875453495699328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575253.846733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495699328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575253.846739:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575253.846742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575253.846744:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575253.846755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699456:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575253.846771:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575253.846778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff93a25064ca80 x1875453495699456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575253.846786:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495699456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575253.846791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575253.846793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575253.846795:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699456:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575253.846799:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495699584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575253.846808:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575253.846814:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff93a25064c700 x1875453495699584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575253.846821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495699584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575253.846826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575253.846827:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575253.846829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495699584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575254.806312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575254.806382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575255.830345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575255.830401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575256.854334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575256.854383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575257.878309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575257.878725:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575258.902583:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575258.902783:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575258.966336:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575258.966344:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575258.966348:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575258.966356:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3475 00000100:00080000:3.0:1788575258.966365:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575258.966370:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575258.966406:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495699712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575258.966437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575258.966440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575258.966445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575258.966449:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3475 00000100:00080000:3.0:1788575258.966452:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575258.966455:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575258.966463:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575258.966474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495699840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575258.966482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575258.966485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575258.966491:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495699712:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575258.966501:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575258.966506:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3475 00000100:00080000:3.0:1788575258.966509:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575258.966512:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575258.966516:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575258.966525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495699968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575258.966533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575258.966536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575258.966537:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575258.966540:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3475 00000100:00080000:3.0:1788575258.966543:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575258.966545:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575258.966548:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575258.966556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495700096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575258.966613:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575258.966664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575258.966691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495700096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575259.926338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575259.926739:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575260.950646:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575260.950728:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575261.974358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575261.975076:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575262.038461:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788575262.038499:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575262.038512:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575262.038556:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575262.038566:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575262.038574:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575262.038666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575262] req@ffff93a25064e300 x1875453495699840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575262.038712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495699840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575262.038730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575262.038737:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575262.038743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575262.038754:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575262] req@ffff93a25064ce00 x1875453495699968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575262.038769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495699968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575262.038779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575262.038784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575262.038788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495699968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575262.038797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575262] req@ffff93a25064f100 x1875453495700096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575262.038811:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495700096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575262.038855:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575262.038861:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575262.038865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495700096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575262.038881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575262] req@ffff93a25064d500 x1875453495699712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575262.038898:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495699712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575262.038908:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575262.038912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575262.038916:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495699712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575262.998476:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575262.998548:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575264.022327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575264.022386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575264.086296:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.086304:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575264.086307:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.086316:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3480 00000100:00080000:3.0:1788575264.086342:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.086348:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575264.086376:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495700224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575264.086397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.086399:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575264.086401:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.086404:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3480 00000100:00080000:3.0:1788575264.086406:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.086408:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575264.086416:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575264.086423:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495700352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575264.086431:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.086433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575264.086435:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.086437:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3480 00000100:00080000:3.0:1788575264.086439:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.086441:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575264.086444:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575264.086454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495700480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575264.086456:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495700224:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575264.086472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.086483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575264.086485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.086489:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3480 00000100:00080000:3.0:1788575264.086494:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.086499:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575264.086505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575264.086518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495700608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575264.086528:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575264.086585:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff93a25064d180 x1875453495700224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575264.086605:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495700224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575264.086617:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575264.086622:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575264.086626:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495700224:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575264.086634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575264.086672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575264.086686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff93a25064c000 x1875453495700352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575264.086701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495700352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575264.086710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575264.086714:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575264.086718:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575264.086745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700480:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575264.086769:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575264.086782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff93a25064f480 x1875453495700480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575264.086795:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495700480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575264.086804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575264.086808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575264.086812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700480:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575264.086819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495700608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575264.086839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575264.086851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff93a25064fb80 x1875453495700608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575264.086889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495700608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575264.086898:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575264.086902:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575264.086906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495700608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575265.046315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575265.046434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575266.070330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575266.070391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575267.094321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575267.094376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575268.118377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575268.118433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575269.142324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575269.142382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575269.206270:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575269.206279:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575269.206282:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575269.206291:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3485 00000100:00080000:3.0:1788575269.206300:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575269.206305:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575269.206338:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495700736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575269.206359:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575269.206362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575269.206363:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575269.206366:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3485 00000100:00080000:3.0:1788575269.206381:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575269.206385:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575269.206395:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575269.206407:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495700736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575269.206413:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495700864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575269.206427:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575269.206431:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575269.206433:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575269.206438:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3485 00000100:00080000:3.0:1788575269.206443:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575269.206447:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575269.206461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575269.206472:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495700992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575269.206481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575269.206485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1788575269.206487:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575269.206488:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575269.206494:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3485 00000100:00080000:3.0:1788575269.206499:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575269.206502:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575269.206505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575269.206513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495701120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575269.206595:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff93a25064e680 x1875453495700736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575269.206627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495700736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575269.206639:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575269.206642:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575269.206646:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495700736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575269.206679:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575269.206699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575269.206708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff93a25064ea00 x1875453495700864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575269.206717:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495700864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575269.206734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575269.206736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575269.206739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575269.206743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700992:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575269.206759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575269.206766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff93a25064d880 x1875453495700992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575269.206795:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495700992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575269.206801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575269.206803:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575269.206805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495700992:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575269.206810:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495701120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575269.206824:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575269.206832:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff93a25064dc00 x1875453495701120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575269.206840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495701120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575269.206845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575269.206846:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575269.206853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495701120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575270.166322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575270.166406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575271.190353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575271.190431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575272.214344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575272.214402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575273.238301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575273.238358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575274.262310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575274.262363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575274.326352:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575274.326362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575274.326366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575274.326376:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3490 00000100:00080000:0.0:1788575274.326385:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575274.326391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575274.326425:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495701248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575274.326449:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575274.326452:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575274.326468:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575274.326473:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3490 00000100:00080000:0.0:1788575274.326477:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575274.326481:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575274.326492:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575274.326496:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495701248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575274.326550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495701376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575274.326561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575274.326564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575274.326566:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575274.326572:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3490 00000800:00000100:1.0:1788575274.326572:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575274.326579:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575274.326583:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575274.326588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575274.326600:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495701504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575274.326610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575274.326625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575274.326628:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575274.326629:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff93a2460d0380 x1875453495701248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.326634:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3490 00000100:00080000:0.0:1788575274.326649:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575274.326653:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495701248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.326657:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575274.326666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788575274.326670:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575274.326677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575274.326682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495701248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575274.326687:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495701632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575274.326715:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495701376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575274.326739:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575274.326752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff93a2460d2680 x1875453495701376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575274.326763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495701376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575274.326771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575274.326773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575274.326775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495701376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575274.326779:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495701504:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575274.326792:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575274.326811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff93a2460d1500 x1875453495701504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575274.326822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495701504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575274.326829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575274.326832:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575274.326835:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495701504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575274.326841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495701632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575274.326855:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575274.326862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff93a2460d3480 x1875453495701632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575274.326869:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495701632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575274.326873:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575274.326875:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575274.326877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495701632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575275.286348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575275.286408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575276.310310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575276.310392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575277.334318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575277.334385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575278.358298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575278.358371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575279.382321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575279.382377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575279.446327:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.446337:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575279.446341:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.446351:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3495 00000100:00080000:2.0:1788575279.446362:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.446368:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575279.446404:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495701760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.446429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.446437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.446439:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.446442:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3495 00000100:00080000:2.0:1788575279.446445:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.446471:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575279.446477:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495701760:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788575279.446481:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575279.446498:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495701888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.446509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.446513:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.446514:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.446518:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3495 00000100:00080000:2.0:1788575279.446523:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575279.446531:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575279.446535:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575279.446540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575279.446551:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495702016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.446558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.446561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.446563:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575279.446564:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff93a249408a80 x1875453495701760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575279.446568:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3495 00000100:00080000:2.0:1788575279.446573:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.446577:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575279.446581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575279.446587:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495701760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575279.446592:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495702144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575279.446621:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.446625:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575279.446629:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495701760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575279.446646:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495701888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575279.446661:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575279.446668:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff93a249409180 x1875453495701888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.446675:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495701888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.446680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.446682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575279.446684:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495701888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575279.446687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702016:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575279.446696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575279.446701:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff93a24940ad80 x1875453495702016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.446707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495702016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.446712:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.446713:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575279.446715:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702016:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575279.446722:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495702144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575279.446734:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575279.446740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff93a24940b800 x1875453495702144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.446746:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495702144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.446750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.446752:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575279.446753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495702144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575280.406317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575280.406367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575281.430306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575281.430309:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575282.454333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575282.454391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575283.478353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575283.478395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575284.502365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575284.502436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575284.566380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575284.566391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575284.566395:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575284.566405:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3501 00000100:00080000:0.0:1788575284.566416:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575284.566423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575284.566462:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495702272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575284.566489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575284.566492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575284.566494:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575284.566499:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3501 00000100:00080000:0.0:1788575284.566502:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575284.566506:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575284.566517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575284.566530:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495702400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575284.566539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575284.566542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575284.566543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575284.566547:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495702272:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575284.566565:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3501 00000100:00080000:0.0:1788575284.566568:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575284.566571:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575284.566577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575284.566594:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495702528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575284.566607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575284.566610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575284.566611:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575284.566615:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3501 00000100:00080000:0.0:1788575284.566617:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575284.566621:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575284.566624:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788575284.566625:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575284.566634:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495702656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575284.566672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff93a2460d3b80 x1875453495702272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.566691:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495702272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.566699:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.566702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575284.566705:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495702272:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575284.566728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575284.566744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575284.566752:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff93a2460d0700 x1875453495702400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.566766:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495702400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.566772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.566774:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575284.566775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575284.566779:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702528:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575284.566791:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575284.566797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff93a2460d0a80 x1875453495702528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.566804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495702528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.566808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.566809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575284.566811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575284.566814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495702656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575284.566823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575284.566828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff93a2460d0000 x1875453495702656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.566833:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495702656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.566845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.566847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575284.566849:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495702656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575285.526326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575285.526403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575286.550350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575286.550421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575287.574330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575287.574393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575288.598313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575288.598370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575289.622325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575289.622388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575289.686368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575289.686376:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575289.686380:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575289.686389:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3506 00000100:00080000:1.0:1788575289.686399:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575289.686406:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575289.686438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495702784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575289.686466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575289.686469:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575289.686470:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575289.686473:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3506 00000100:00080000:1.0:1788575289.686475:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575289.686478:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575289.686485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575289.686494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495702912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575289.686501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575289.686504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575289.686516:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575289.686521:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3506 00000100:00080000:1.0:1788575289.686525:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575289.686528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495702784:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575289.686531:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575289.686542:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575289.686558:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495703040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575289.686571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575289.686575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575289.686577:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575289.686582:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3506 00000100:00080000:1.0:1788575289.686586:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575289.686590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575289.686593:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575289.686601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575289.686616:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495703168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575289.686641:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff93a258da2300 x1875453495702784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575289.686667:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495702784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575289.686676:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575289.686679:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575289.686682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495702784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575289.686687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575289.686705:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575289.686718:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff93a258da0000 x1875453495702912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575289.686731:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495702912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575289.686740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575289.686743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575289.686747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495702912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575289.686771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703040:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575289.686795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575289.686808:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff93a258da0380 x1875453495703040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575289.686821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495703040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575289.686836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575289.686839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575289.686842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703040:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575289.686846:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495703168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575289.686862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575289.686879:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff93a258da1180 x1875453495703168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575289.686887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495703168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575289.686892:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575289.686894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575289.686896:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495703168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575290.646292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575290.646359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575291.670327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575291.670388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575292.694299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575292.694375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575293.718319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575293.718381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575294.742376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575294.742432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575294.806391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575294.806403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575294.806407:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575294.806419:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3511 00000100:00080000:0.0:1788575294.806430:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575294.806438:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575294.806500:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495703296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575294.806528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575294.806531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575294.806533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575294.806536:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3511 00000100:00080000:0.0:1788575294.806539:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575294.806542:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575294.806553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575294.806562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495703424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575294.806570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575294.806572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495703296:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575294.806573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575294.806577:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575294.806584:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3511 00000100:00080000:0.0:1788575294.806586:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575294.806588:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575294.806591:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575294.806598:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495703552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575294.806605:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575294.806607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575294.806608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575294.806617:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3511 00000100:00080000:0.0:1788575294.806620:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575294.806624:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575294.806629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575294.806641:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495703680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575294.806664:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575294.806714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff93a2460d0000 x1875453495703296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.806728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495703296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.806737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.806746:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575294.806750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495703296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575294.806779:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575294.806801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575294.806811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff93a2460d0a80 x1875453495703424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.806819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495703424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.806824:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.806826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575294.806829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575294.806833:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703552:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575294.806845:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575294.806851:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff93a2460d0700 x1875453495703552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.806858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495703552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.806862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.806864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575294.806865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703552:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575294.806868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495703680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575294.806885:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575294.806895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff93a2460d3b80 x1875453495703680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.806905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495703680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.806913:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.806916:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575294.806919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495703680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575295.766317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575295.766368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575296.790354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575296.790413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575297.814319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575297.814379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575298.838331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575298.838395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575299.862286:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575299.862386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575299.926402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575299.926410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575299.926413:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575299.926423:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3516 00000100:00080000:1.0:1788575299.926431:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575299.926437:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575299.926470:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495703808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575299.926510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575299.926514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575299.926517:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575299.926522:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3516 00000100:00080000:1.0:1788575299.926527:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575299.926540:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575299.926552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575299.926571:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495703808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575299.926581:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495703936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575299.926592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575299.926594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575299.926596:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575299.926600:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3516 00000100:00080000:1.0:1788575299.926605:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575299.926610:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575299.926617:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575299.926631:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495704064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575299.926642:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575299.926647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575299.926649:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575299.926653:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3516 00000800:00000100:0.0:1788575299.926658:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575299.926659:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575299.926670:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575299.926675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575299.926688:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495704192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575299.926724:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff93a258da3480 x1875453495703808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575299.926742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495703808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575299.926751:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575299.926755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575299.926758:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495703808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575299.926781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575299.926797:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575299.926821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff93a258da2a00 x1875453495703936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575299.926844:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495703936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575299.926850:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575299.926852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575299.926854:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495703936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575299.926859:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704064:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575299.926875:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575299.926881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff93a258da3100 x1875453495704064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575299.926889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495704064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575299.926893:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575299.926895:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575299.926896:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575299.926900:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495704192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575299.926911:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575299.926917:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff93a258da1880 x1875453495704192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575299.926923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495704192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575299.926936:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575299.926938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575299.926940:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495704192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575300.886341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575300.886403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575301.910339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575301.910408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575302.934317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575302.934384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575303.958316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575303.958375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575304.982325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575304.982372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575305.046349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575305.046361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575305.046365:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575305.046391:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3521 00000100:00080000:1.0:1788575305.046402:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575305.046409:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575305.046443:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495704320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575305.046472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575305.046476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575305.046478:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575305.046483:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3521 00000100:00080000:1.0:1788575305.046486:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575305.046491:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575305.046501:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575305.046516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495704320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575305.046517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495704448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575305.046532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575305.046535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575305.046538:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575305.046542:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3521 00000100:00080000:1.0:1788575305.046546:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575305.046551:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575305.046556:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575305.046567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495704576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575305.046577:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575305.046596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575305.046600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575305.046603:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575305.046608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3521 00000100:00080000:1.0:1788575305.046613:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575305.046618:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff93a258da3800 x1875453495704320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575305.046621:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575305.046630:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575305.046658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495704320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575305.046662:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495704704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575305.046684:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.046692:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575305.046697:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495704320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575305.046724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575305.046751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575305.046767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff93a258da1c00 x1875453495704448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.046779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495704448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.046794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.046798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575305.046801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575305.046807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704576:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575305.046830:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575305.046844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff93a258da0700 x1875453495704576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.046868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495704576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.046878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.046881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575305.046885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704576:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575305.046890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495704704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575305.046907:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575305.046916:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff93a258da0e00 x1875453495704704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.046923:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495704704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.046929:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.046931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575305.046939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495704704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575306.006320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575306.006381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575307.030335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575307.030386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575308.054379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575308.054432:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575309.078326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575309.078380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575310.102340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575310.102395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575310.166353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.166365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575310.166369:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.166380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3526 00000100:00080000:3.0:1788575310.166393:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.166401:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575310.166457:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495704832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575310.166498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.166503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575310.166506:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.166511:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3526 00000100:00080000:3.0:1788575310.166516:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.166521:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575310.166534:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575310.166550:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495704960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575310.166556:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495704832:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575310.166585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.166588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575310.166590:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.166594:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3526 00000100:00080000:3.0:1788575310.166598:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.166602:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575310.166606:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575310.166616:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495705088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575310.166623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.166626:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575310.166627:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575310.166636:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575310.166643:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3526 00000100:00080000:3.0:1788575310.166647:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.166650:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575310.166654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575310.166663:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495705216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575310.166710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff93a25064ed80 x1875453495704832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.166735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495704832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.166748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.166753:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575310.166758:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495704832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575310.166787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575310.166806:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575310.166816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff93a25064dc00 x1875453495704960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.166825:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495704960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.166831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.166833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575310.166842:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495704960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575310.166847:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705088:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575310.166863:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575310.166870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff93a25064d880 x1875453495705088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.166878:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495705088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.166883:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.166885:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575310.166888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705088:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575310.166891:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495705216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575310.166905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575310.166915:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff93a25064ea00 x1875453495705216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.166928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495705216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.166937:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.166940:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575310.166944:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495705216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575311.126323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575311.126373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575312.150293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575312.150352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575313.174342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575313.174416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575314.198330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575314.198387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575315.222359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575315.222416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575315.286412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575315.286423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575315.286428:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575315.286439:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3531 00000100:00080000:1.0:1788575315.286452:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575315.286460:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575315.286503:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495705344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575315.286537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575315.286542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575315.286544:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575315.286549:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3531 00000100:00080000:1.0:1788575315.286554:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575315.286559:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575315.286589:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495705344:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788575315.286591:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575315.286612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495705472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575315.286625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575315.286629:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575315.286631:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575315.286638:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3531 00000100:00080000:1.0:1788575315.286659:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575315.286664:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788575315.286671:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788575315.286673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575315.286690:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495705600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575315.286699:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575315.286703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575315.286704:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575315.286707:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3531 00000100:00080000:1.0:1788575315.286710:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575315.286713:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575315.286717:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575315.286727:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495705728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575315.286734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff93a258da2680 x1875453495705344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.286755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495705344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.286766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.286770:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575315.286773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495705344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575315.286798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575315.286818:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575315.286828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff93a258da0a80 x1875453495705472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.286839:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495705472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.286848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.286852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575315.286855:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575315.286863:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705600:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575315.286900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575315.286910:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff93a258da1f80 x1875453495705600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.286918:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495705600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.286924:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.286927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575315.286929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575315.286933:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495705728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575315.286946:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575315.286957:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff93a258da0e00 x1875453495705728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.286971:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495705728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.286980:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.286984:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575315.286987:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495705728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575316.246373:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575316.246436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575317.270318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575317.270411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575318.294370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575318.294457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575319.318298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575319.318373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575320.342375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575320.342420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575320.406310:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575320.406319:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575320.406322:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575320.406330:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3536 00000100:00080000:2.0:1788575320.406338:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575320.406344:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575320.406377:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495705856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575320.406399:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575320.406401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575320.406403:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575320.406406:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3536 00000100:00080000:2.0:1788575320.406408:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575320.406410:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575320.406418:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575320.406427:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495705984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575320.406435:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575320.406439:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788575320.406440:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495705856:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575320.406440:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575320.406444:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3536 00000100:00080000:2.0:1788575320.406446:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575320.406448:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575320.406461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575320.406471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495706112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575320.406479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575320.406481:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575320.406483:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575320.406486:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3536 00000100:00080000:2.0:1788575320.406488:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575320.406491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575320.406495:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575320.406502:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495706240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575320.406518:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575320.406560:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff93a24940a680 x1875453495705856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.406575:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495705856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.406583:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.406587:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575320.406590:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495705856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575320.406608:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575320.406624:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575320.406632:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff93a249409c00 x1875453495705984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.406640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495705984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.406645:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.406647:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575320.406650:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495705984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575320.406653:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495706112:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575320.406665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575320.406671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff93a249408e00 x1875453495706112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.406689:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495706112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.406694:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.406697:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575320.406699:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495706112:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575320.406702:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495706240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575320.406715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575320.406722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff93a24940b800 x1875453495706240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.406729:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495706240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.406733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.406735:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575320.406737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495706240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575321.366325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575321.366379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575322.390334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575322.390387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575323.414320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575323.414368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575324.438342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575324.438374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575325.462507:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575325.462608:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575325.526313:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575325.526321:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575325.526325:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575325.526349:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3541 00000100:00080000:0.0:1788575325.526357:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575325.526364:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575325.526396:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495706368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575325.526429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575325.526438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575325.526441:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575325.526446:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3541 00000100:00080000:0.0:1788575325.526451:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575325.526457:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575325.526468:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575325.526477:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495706368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575325.526486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495706496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575325.526503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575325.526507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575325.526509:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575325.526514:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3541 00000100:00080000:0.0:1788575325.526519:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575325.526524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575325.526530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575325.526545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495706624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575325.526571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575325.526575:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575325.526579:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575325.526583:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495706496:192.168.202.102@tcp:38:kworker.0 00000100:00080000:0.0:1788575325.526588:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3541 00000100:00080000:0.0:1788575325.526596:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575325.526602:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575325.526609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575325.526626:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495706752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575325.526629:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495706624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575325.526664:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495706752:192.168.202.102@tcp:38:kworker.0 00010000:00010000:3.0:1788575326.486951:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575326.487223:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575327.510487:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575327.510953:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575328.534597:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575328.535294:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788575328.598528:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788575328.598581:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575328.598599:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575328.598645:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575328.598655:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575328.598664:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575328.598751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff93a2460d0380 x1875453495706752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575328.598776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495706752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575328.598792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575328.598798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575328.598802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495706752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575328.598811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff93a2460d1500 x1875453495706496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575328.598823:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495706496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575328.598830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575328.598834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575328.598837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495706496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575328.598844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff93a2460d2680 x1875453495706624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575328.598855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495706624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575328.598878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575328.598882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575328.598886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495706624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575328.598906:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff93a2460d3480 x1875453495706368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575328.598919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495706368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575328.598927:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575328.598930:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575328.598933:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495706368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575329.558345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575329.558546:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575330.582324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575330.582422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575330.646451:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.646467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575330.646472:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.646487:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3547 00000100:00080000:3.0:1788575330.646501:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.646510:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575330.646558:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495706880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575330.646613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.646617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575330.646619:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.646625:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3547 00000100:00080000:3.0:1788575330.646630:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.646635:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575330.646647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575330.646663:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495707008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575330.646674:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575330.646675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495706880:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575330.646677:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575330.646681:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.646685:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3547 00000100:00080000:3.0:1788575330.646689:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.646693:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575330.646698:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575330.646707:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495707136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575330.646717:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.646720:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575330.646722:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.646726:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3547 00000100:00080000:3.0:1788575330.646737:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.646741:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575330.646746:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575330.646757:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495707264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575330.646763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575330.646818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff93a25064e680 x1875453495706880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.646872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495706880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.646886:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.646906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575330.646911:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495706880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575330.646951:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575330.646993:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575330.647011:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff93a25064fb80 x1875453495707008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.647025:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495707008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.647034:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.647039:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575330.647043:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575330.647049:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707136:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575330.647078:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575330.647091:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff93a25064f480 x1875453495707136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.647103:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495707136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.647112:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.647115:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575330.647118:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707136:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575330.647124:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495707264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575330.647146:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575330.647159:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff93a25064c000 x1875453495707264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.647187:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495707264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.647196:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.647200:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575330.647203:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495707264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575331.606323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575331.606385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575332.630317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575332.630361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575333.654325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575333.654400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575334.678295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575334.678365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575335.702320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575335.702368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575335.766396:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.766407:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575335.766411:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.766427:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3552 00000100:00080000:3.0:1788575335.766437:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.766446:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575335.766484:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495707392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575335.766549:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495707392:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788575335.766627:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575335.766660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.766664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575335.766666:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.766671:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3552 00000100:00080000:3.0:1788575335.766676:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.766680:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575335.766690:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575335.766718:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495707520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575335.766807:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff93a25064d180 x1875453495707392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575335.766826:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495707392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575335.766836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575335.766840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575335.766845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495707392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575335.766870:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575335.766894:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575335.766908:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff93a25064d500 x1875453495707520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575335.766919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495707520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575335.766940:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575335.766944:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575335.766947:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707520:192.168.202.102@tcp:38:kworker.0 00000100:00080000:3.0:1788575335.767000:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.767004:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575335.767007:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.767012:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3552 00000100:00080000:3.0:1788575335.767017:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.767021:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575335.767028:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575335.767042:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495707648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575335.767053:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.767057:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575335.767058:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.767062:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3552 00000100:00080000:3.0:1788575335.767066:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.767070:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575335.767075:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575335.767085:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495707776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575335.767156:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707648:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575335.767182:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575335.767196:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff93a25064f100 x1875453495707648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575335.767209:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495707648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575335.767219:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575335.767222:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575335.767225:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495707648:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575335.767253:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495707776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575335.767320:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575335.767332:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff93a25064ce00 x1875453495707776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575335.767345:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495707776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575335.767353:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575335.767356:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575335.767367:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495707776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575336.726309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575336.726403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575337.750334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575337.750394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575338.774298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575338.774365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575339.798367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575339.798412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575340.822354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575340.822452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575340.886385:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575340.886400:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575340.886405:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575340.886419:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3557 00000100:00080000:3.0:1788575340.886432:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575340.886442:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575340.886490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495707904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575340.886521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575340.886525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575340.886526:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575340.886529:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3557 00000100:00080000:3.0:1788575340.886532:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575340.886536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575340.886547:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575340.886557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495708032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575340.886566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575340.886568:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575340.886569:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575340.886576:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3557 00000100:00100000:1.0:1788575340.886578:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495707904:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575340.886578:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575340.886583:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575340.886613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575340.886625:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495708160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575340.886635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575340.886638:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575340.886639:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575340.886642:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3557 00000100:00080000:3.0:1788575340.886646:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575340.886649:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575340.886652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575340.886660:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575340.886662:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495708288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575340.886742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575340] req@ffff93a25064e300 x1875453495707904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575340.886776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495707904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575340.886788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575340.886794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575340.886798:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495707904:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575340.886831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575340.886859:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575340.886875:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575340] req@ffff93a25064c700 x1875453495708032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575340.886891:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495708032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575340.886902:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575340.886906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575340.886910:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575340.886917:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708160:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575340.886942:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575340.886964:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575340] req@ffff93a25064ca80 x1875453495708160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575340.886974:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495708160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575340.886980:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575340.886983:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575340.886985:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575340.886990:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495708288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575340.887006:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575340.887013:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575340] req@ffff93a25064c380 x1875453495708288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575340.887021:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495708288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575340.887027:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575340.887029:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575340.887031:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495708288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575341.846355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575341.846435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575342.870367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575342.870419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575343.894393:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575343.894431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575344.918318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575344.918360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575345.942369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575345.942431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575346.006399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.006413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575346.006419:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.006433:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3562 00000100:00080000:3.0:1788575346.006447:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.006456:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575346.006506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495708416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575346.006536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.006540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575346.006560:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.006567:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3562 00000100:00100000:1.0:1788575346.006574:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495708416:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575346.006574:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.006583:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575346.006613:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575346.006633:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495708544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575346.006647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.006651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575346.006653:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.006659:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3562 00000100:00080000:3.0:1788575346.006663:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.006667:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575346.006673:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575346.006686:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495708672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575346.006693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575346.006703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.006708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575346.006711:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.006717:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3562 00000100:00080000:3.0:1788575346.006721:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.006726:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575346.006731:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575346.006750:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495708800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575346.006764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff93a25064df80 x1875453495708416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575346.006789:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495708416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575346.006810:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575346.006815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575346.006821:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495708416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575346.006853:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575346.006879:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575346.006892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff93a25064c380 x1875453495708544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575346.006905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495708544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575346.006915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575346.006918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575346.006922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575346.006929:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708672:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575346.006953:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575346.006964:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff93a25064ca80 x1875453495708672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575346.006977:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495708672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575346.006985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575346.006989:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575346.006993:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495708672:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575346.007005:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495708800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575346.007040:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575346.007052:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff93a25064c700 x1875453495708800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575346.007065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495708800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575346.007074:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575346.007078:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575346.007082:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495708800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575346.966314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575346.966358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575347.990323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575347.990403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575349.014327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575349.014427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575350.038320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575350.038416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575351.062322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575351.062375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575351.126312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575351.126321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575351.126324:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575351.126333:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3567 00000100:00080000:2.0:1788575351.126342:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575351.126348:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575351.126379:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495708928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575351.126400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575351.126403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575351.126404:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575351.126407:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3567 00000100:00080000:2.0:1788575351.126410:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575351.126412:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575351.126419:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575351.126433:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495709056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575351.126453:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495708928:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575351.126464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575351.126468:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575351.126469:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575351.126473:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3567 00000100:00080000:2.0:1788575351.126476:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575351.126479:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575351.126483:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575351.126493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495709184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575351.126500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575351.126503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575351.126504:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575351.126507:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3567 00000100:00080000:2.0:1788575351.126509:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575351.126528:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575351.126532:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575351.126542:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575351.126543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495709312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575351.126595:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff93a24940ad80 x1875453495708928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575351.126618:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495708928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575351.126630:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575351.126635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575351.126640:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495708928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575351.126668:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575351.126688:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575351.126697:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff93a249409180 x1875453495709056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575351.126713:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495709056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575351.126719:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575351.126721:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575351.126724:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575351.126728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709184:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575351.126742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575351.126749:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff93a249408a80 x1875453495709184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575351.126756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495709184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575351.126761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575351.126763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575351.126765:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575351.126769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495709312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575351.126779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575351.126785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff93a249408700 x1875453495709312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575351.126793:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495709312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575351.126807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575351.126809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575351.126812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495709312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575352.086294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575352.086382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575353.110356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575353.110363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575354.134303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575354.134365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575355.158333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575355.158396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575356.182292:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575356.182456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575356.246341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575356.246350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575356.246353:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575356.246363:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3572 00000100:00080000:0.0:1788575356.246393:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575356.246401:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575356.246432:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495709440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575356.246469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575356.246472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575356.246474:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575356.246477:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3572 00000100:00080000:0.0:1788575356.246480:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575356.246483:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575356.246492:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575356.246503:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495709568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575356.246512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575356.246517:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495709440:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575356.246518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575356.246523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575356.246530:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3572 00000100:00080000:0.0:1788575356.246534:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575356.246538:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575356.246544:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575356.246559:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495709696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575356.246570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575356.246575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575356.246581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575356.246583:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575356.246588:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3572 00000100:00080000:0.0:1788575356.246592:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575356.246595:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575356.246600:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575356.246613:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495709824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575356.246613:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff93a2460d2300 x1875453495709440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575356.246656:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495709440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575356.246665:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575356.246669:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575356.246673:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495709440:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575356.246691:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575356.246706:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575356.246714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff93a2460d1f80 x1875453495709568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575356.246722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495709568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575356.246728:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575356.246730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575356.246732:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575356.246743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709696:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575356.246758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575356.246765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff93a2460d2a00 x1875453495709696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575356.246773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495709696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575356.246778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575356.246780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575356.246781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495709696:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575356.246785:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495709824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575356.246796:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575356.246802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff93a2460d1180 x1875453495709824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575356.246809:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495709824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575356.246814:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575356.246816:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575356.246818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495709824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575357.206339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575357.206402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575358.230333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575358.230478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575359.254318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575359.254430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575360.278315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575360.278367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575361.302314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575361.302479:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575361.366312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575361.366320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575361.366324:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575361.366331:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3577 00000100:00080000:2.0:1788575361.366340:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575361.366345:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575361.366394:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495709952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575361.366415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575361.366418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575361.366419:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575361.366422:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3577 00000100:00080000:2.0:1788575361.366425:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575361.366427:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575361.366435:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575361.366448:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495710080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575361.366459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575361.366463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575361.366466:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575361.366487:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3577 00000100:00080000:2.0:1788575361.366492:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575361.366495:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575361.366500:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575361.366514:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495709952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575361.366514:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495710208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575361.366545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575361.366548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575361.366549:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575361.366552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3577 00000100:00080000:2.0:1788575361.366555:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575361.366558:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575361.366560:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575361.366571:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495710336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575361.366621:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575361.366679:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff93a249409880 x1875453495709952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575361.366713:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495709952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575361.366724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575361.366728:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575361.366731:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495709952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575361.366741:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575361.366768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575361.366783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff93a249408000 x1875453495710080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575361.366795:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495710080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575361.366801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575361.366804:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575361.366806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575361.366837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710208:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575361.366861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575361.366874:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff93a24940aa00 x1875453495710208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575361.366885:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495710208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575361.366901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575361.366904:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575361.366906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710208:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575361.366911:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495710336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575361.366928:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575361.366936:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff93a249409500 x1875453495710336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575361.366944:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495710336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575361.366950:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575361.366952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575361.366954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495710336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575362.326311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575362.326374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575363.350344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575363.350401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575364.374345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575364.374408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575365.398352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575365.398405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575366.422339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575366.422464:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575366.486332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575366.486341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575366.486352:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575366.486360:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3582 00000100:00080000:0.0:1788575366.486369:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575366.486375:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575366.486408:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495710464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575366.486429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575366.486432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575366.486446:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575366.486450:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3582 00000100:00080000:0.0:1788575366.486453:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575366.486456:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575366.486471:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575366.486482:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495710592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575366.486491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575366.486494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575366.486496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575366.486505:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3582 00000100:00100000:1.0:1788575366.486508:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495710464:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575366.486509:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575366.486513:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575366.486523:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575366.486533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495710720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575366.486540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575366.486542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575366.486543:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575366.486546:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3582 00000100:00080000:0.0:1788575366.486550:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575366.486552:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575366.486555:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575366.486568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495710848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575366.486616:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575366.486674:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff93a2460d1180 x1875453495710464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575366.486697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495710464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575366.486711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575366.486716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575366.486722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495710464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575366.486771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575366.486812:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575366.486827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff93a2460d2a00 x1875453495710592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575366.486841:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495710592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575366.486851:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575366.486855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575366.486867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575366.486875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710720:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575366.486900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575366.486912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff93a2460d1f80 x1875453495710720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575366.486926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495710720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575366.486935:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575366.486939:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575366.486942:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495710720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575366.486949:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495710848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575366.486988:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575366.487001:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff93a2460d2300 x1875453495710848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575366.487037:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495710848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575366.487047:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575366.487051:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575366.487054:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495710848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575367.446339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575367.446381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575368.470336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575368.470373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575369.494309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575369.494361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575370.518310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575370.518359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575371.542336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575371.542392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575371.606330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575371.606338:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575371.606341:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575371.606350:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3588 00000100:00080000:0.0:1788575371.606359:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575371.606365:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575371.606393:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495710976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575371.606415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575371.606418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575371.606419:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575371.606422:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3588 00000100:00080000:0.0:1788575371.606425:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575371.606428:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575371.606436:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575371.606450:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495711104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575371.606450:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495710976:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575371.606465:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575371.606467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575371.606468:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575371.606470:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3588 00000100:00080000:0.0:1788575371.606473:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575371.606489:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575371.606493:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575371.606502:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495711232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575371.606510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575371.606513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575371.606514:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575371.606517:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3588 00000100:00080000:0.0:1788575371.606520:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575371.606521:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575371.606523:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575371.606528:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575371.606535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495711360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575371.606561:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff93a2460d3480 x1875453495710976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.606577:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495710976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.606585:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.606588:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575371.606591:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495710976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575371.606610:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575371.606627:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575371.606635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff93a2460d2680 x1875453495711104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.606643:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495711104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.606659:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.606662:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575371.606664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575371.606669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711232:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575371.606684:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575371.606691:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff93a2460d1500 x1875453495711232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.606700:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495711232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.606709:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.606712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575371.606715:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711232:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575371.606722:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495711360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575371.606753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575371.606764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff93a2460d0380 x1875453495711360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.606773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495711360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.606778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.606780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575371.606788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495711360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575372.566336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575372.566405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575373.590321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575373.590377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575374.614335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575374.614400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575375.638355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575375.638424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575376.662325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575376.662393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575376.726334:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575376.726345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575376.726350:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575376.726361:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3593 00000100:00080000:0.0:1788575376.726373:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575376.726381:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575376.726424:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495711488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575376.726453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575376.726456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575376.726473:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575376.726477:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3593 00000100:00080000:0.0:1788575376.726481:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575376.726485:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575376.726496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575376.726508:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495711616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575376.726517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575376.726520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575376.726522:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575376.726529:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3593 00000100:00100000:1.0:1788575376.726531:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495711488:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575376.726533:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575376.726544:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575376.726551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575376.726562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495711744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575376.726570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575376.726573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575376.726574:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575376.726577:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3593 00000100:00080000:0.0:1788575376.726580:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575376.726582:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575376.726586:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575376.726600:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495711872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575376.726610:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575376.726662:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff93a2460d3b80 x1875453495711488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.726677:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495711488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.726685:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.726689:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575376.726692:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495711488:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575376.726715:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575376.726740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575376.726751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff93a2460d0700 x1875453495711616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.726760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495711616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.726765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.726768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575376.726770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575376.726774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711744:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575376.726788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575376.726810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff93a2460d0a80 x1875453495711744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.726817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495711744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.726823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.726825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575376.726827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495711744:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575376.726831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495711872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575376.726843:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575376.726849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff93a2460d0000 x1875453495711872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.726862:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495711872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.726867:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.726869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575376.726871:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495711872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575377.686318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575377.686368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575378.710352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575378.710417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575379.734391:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575379.734459:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575380.758379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575380.758470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575381.782356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575381.782440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575381.846382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575381.846394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575381.846399:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575381.846410:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3598 00000100:00080000:2.0:1788575381.846422:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575381.846429:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575381.846470:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495712000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575381.846496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575381.846499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575381.846500:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575381.846504:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3598 00000100:00080000:2.0:1788575381.846507:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575381.846509:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575381.846518:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575381.846527:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495712128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575381.846561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575381.846564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575381.846565:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575381.846569:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3598 00000100:00080000:2.0:1788575381.846572:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575381.846575:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575381.846580:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575381.846583:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495712000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575381.846591:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495712256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575381.846601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575381.846604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575381.846605:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575381.846608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3598 00000100:00080000:2.0:1788575381.846611:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575381.846613:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575381.846615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575381.846626:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495712384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575381.846660:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575381.846726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff93a24940b100 x1875453495712000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575381.846745:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495712000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575381.846761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575381.846765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575381.846768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495712000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575381.846774:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575381.846793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575381.846805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff93a24940bb80 x1875453495712128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575381.846818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495712128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575381.846827:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575381.846831:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575381.846834:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575381.846865:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712256:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575381.846886:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575381.846895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff93a24940b480 x1875453495712256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575381.846903:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495712256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575381.846909:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575381.846911:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575381.846913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575381.846926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495712384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575381.846940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575381.846947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff93a249409500 x1875453495712384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575381.846955:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495712384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575381.846960:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575381.846962:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575381.846964:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495712384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575382.806354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575382.806465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575383.830370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575383.830428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575384.854352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575384.854423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575385.878340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575385.878414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575386.902362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575386.902444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575386.966377:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575386.966390:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575386.966395:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575386.966406:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3603 00000100:00080000:2.0:1788575386.966426:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575386.966433:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575386.966483:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495712512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575386.966517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575386.966522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575386.966524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575386.966529:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3603 00000100:00080000:2.0:1788575386.966533:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575386.966538:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575386.966550:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575386.966567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495712640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575386.966578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575386.966582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575386.966583:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495712512:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575386.966584:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575386.966588:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3603 00000100:00080000:2.0:1788575386.966592:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575386.966596:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575386.966602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575386.966631:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495712768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575386.966641:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575386.966645:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575386.966646:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575386.966650:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3603 00000100:00080000:2.0:1788575386.966654:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575386.966658:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575386.966663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575386.966672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575386.966675:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495712896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575386.966735:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff93a24940aa00 x1875453495712512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575386.966756:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495712512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575386.966765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575386.966769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575386.966772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495712512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575386.966801:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575386.966821:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575386.966840:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff93a249408000 x1875453495712640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575386.966860:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495712640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575386.966866:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575386.966869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575386.966871:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575386.966875:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712768:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575386.966892:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575386.966899:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff93a249409880 x1875453495712768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575386.966907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495712768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575386.966912:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575386.966913:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575386.966915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495712768:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575386.966919:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495712896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575386.966930:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575386.966936:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff93a249408700 x1875453495712896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575386.966944:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495712896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575386.966958:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575386.966994:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575386.966997:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495712896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575387.926343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575387.926397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575388.950298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575388.950379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575389.974309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575389.974379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575390.998334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575390.998399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575392.022466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575392.022856:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788575392.086392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575392.086404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575392.086409:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575392.086421:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3608 00000100:00080000:0.0:1788575392.086433:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575392.086442:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575392.086483:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495713024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575392.086515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575392.086519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575392.086521:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575392.086526:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3608 00000100:00080000:0.0:1788575392.086546:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575392.086552:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575392.086571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575392.086585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495713152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575392.086596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575392.086599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575392.086602:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575392.086607:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3608 00000100:00080000:0.0:1788575392.086610:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575392.086615:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575392.086621:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495713024:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788575392.086623:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575392.086640:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495713280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575392.086652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575392.086657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575392.086659:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575392.086664:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3608 00000100:00080000:0.0:1788575392.086687:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575392.086691:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575392.086696:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575392.086708:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495713408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575392.086731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575392.086784:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575392.086814:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495713408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575393.046607:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575393.047190:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575394.070513:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575394.070626:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575395.094426:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575395.094684:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788575395.158545:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:0.0:1788575395.158594:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575395.158612:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788575395.158663:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788575395.158675:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:0.0:1788575395.158683:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575395.158729:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff93a2460d0700 x1875453495713280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.158758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495713280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.158771:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.158777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575395.158782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575395.158792:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff93a2460d3b80 x1875453495713408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.158806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495713408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.158842:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.158864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575395.158868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495713408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575395.158876:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff93a2460d0000 x1875453495713024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.158892:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495713024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.158900:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.158904:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575395.158907:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495713024:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575395.158919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff93a2460d0a80 x1875453495713152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.158933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495713152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.158941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.158945:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575395.158948:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713152:192.168.202.102@tcp:38:kworker.0 00010000:00010000:1.0:1788575396.118585:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575396.118735:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575397.142404:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575397.142413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575397.206428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.206439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575397.206443:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.206455:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3613 00000100:00080000:2.0:1788575397.206466:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.206480:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575397.206517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495713536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575397.206546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.206550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575397.206581:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.206589:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3613 00000100:00100000:1.0:1788575397.206598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495713536:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575397.206603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.206618:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575397.206633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575397.206658:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495713664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575397.206672:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.206678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575397.206680:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.206686:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3613 00000100:00080000:2.0:1788575397.206691:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.206696:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575397.206702:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575397.206717:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495713792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575397.206724:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575397.206734:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.206739:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575397.206740:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.206751:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3613 00000100:00080000:2.0:1788575397.206757:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.206762:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575397.206770:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575397.206802:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495713920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575397.206920:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff93a249408a80 x1875453495713536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.206947:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495713536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.206961:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.206975:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575397.206980:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495713536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575397.207022:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575397.207052:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575397.207069:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff93a249409180 x1875453495713664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.207083:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495713664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.207092:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.207095:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575397.207099:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575397.207106:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713792:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575397.207129:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575397.207142:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff93a24940ad80 x1875453495713792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.207156:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495713792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.207178:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.207183:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575397.207187:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495713792:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575397.207195:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495713920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575397.207222:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575397.207274:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff93a24940b800 x1875453495713920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.207287:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495713920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.207296:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.207299:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575397.207303:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495713920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575398.166324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575398.166370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575399.190337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575399.190400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575400.214362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575400.214409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575401.238346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575401.238422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575402.262302:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575402.262356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575402.326316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.326343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575402.326347:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.326357:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3618 00000100:00080000:3.0:1788575402.326367:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.326376:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575402.326423:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495714048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575402.326453:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.326457:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575402.326459:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.326463:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3618 00000100:00080000:3.0:1788575402.326466:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.326470:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575402.326482:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575402.326493:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495714176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575402.326504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.326507:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575402.326508:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.326511:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3618 00000100:00080000:3.0:1788575402.326513:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.326516:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575402.326519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575402.326525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495714304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575402.326539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.326543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575402.326545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575402.326549:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495714048:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575402.326549:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3618 00000100:00080000:3.0:1788575402.326556:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.326564:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575402.326567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575402.326577:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495714432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575402.326621:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575402.326669:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff93a25064e300 x1875453495714048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.326687:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495714048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.326697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.326701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575402.326705:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495714048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575402.326726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575402.326744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575402.326753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff93a25064ce00 x1875453495714176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.326760:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495714176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.326767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.326771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575402.326774:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575402.326829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714304:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575402.326851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575402.326865:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff93a25064f100 x1875453495714304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.326873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495714304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.326878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.326880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575402.326881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714304:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575402.326885:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495714432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575402.326898:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575402.326905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff93a25064d500 x1875453495714432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.326918:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495714432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.326925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.326928:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575402.326931:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495714432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575403.286320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575403.286380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575404.310328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575404.310384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575405.334343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575405.334394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575406.358344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575406.358398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575407.382341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575407.382431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575407.446419:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.446432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575407.446437:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.446450:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3623 00000100:00080000:2.0:1788575407.446463:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.446472:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575407.446529:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495714560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575407.446561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.446565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575407.446567:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.446571:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3623 00000100:00080000:2.0:1788575407.446576:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.446580:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575407.446592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575407.446607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495714688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575407.446618:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.446622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575407.446634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495714560:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575407.446635:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.446650:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3623 00000100:00080000:2.0:1788575407.446655:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.446678:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575407.446685:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575407.446698:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495714816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575407.446709:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.446717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1788575407.446718:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575407.446721:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.446728:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3623 00000100:00080000:2.0:1788575407.446750:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.446756:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575407.446762:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575407.446779:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff93a249408e00 x1875453495714560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575407.446781:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495714944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575407.446828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495714560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.446841:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.446845:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575407.446850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495714560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575407.446877:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575407.446903:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575407.446916:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff93a249409c00 x1875453495714688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.446928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495714688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.446955:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.446959:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575407.446962:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575407.446970:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714816:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575407.446994:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575407.447004:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff93a24940a680 x1875453495714816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.447014:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495714816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.447021:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.447024:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575407.447027:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495714816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575407.447032:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495714944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575407.447047:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575407.447055:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff93a24940b800 x1875453495714944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.447064:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495714944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.447071:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.447073:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575407.447080:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495714944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575408.406362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575408.408209:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575409.430304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575409.430345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575410.454328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575410.454410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575411.478340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575411.478422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575412.502321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575412.502391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575412.566386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.566397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575412.566402:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.566411:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3629 00000100:00080000:2.0:1788575412.566423:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.566430:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575412.566483:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495715072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575412.566510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.566513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575412.566528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.566532:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3629 00000100:00080000:2.0:1788575412.566535:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.566539:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575412.566549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575412.566559:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495715200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575412.566566:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495715072:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575412.566569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.566575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575412.566577:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.566581:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3629 00000100:00080000:2.0:1788575412.566583:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.566585:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575412.566587:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575412.566596:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495715328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575412.566602:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.566605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575412.566606:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.566608:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3629 00000100:00080000:2.0:1788575412.566610:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.566612:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575412.566614:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575412.566627:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495715456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575412.566647:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575412.566703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff93a24940ad80 x1875453495715072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.566723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495715072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.566735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.566740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575412.566744:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495715072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575412.566773:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575412.566795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575412.566823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff93a249409180 x1875453495715200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.566836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495715200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.566845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.566849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575412.566852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575412.566858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715328:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575412.566878:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575412.566895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff93a249408a80 x1875453495715328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.566907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495715328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.566915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.566919:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575412.566922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715328:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575412.566928:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495715456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575412.566945:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575412.566954:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff93a249408700 x1875453495715456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.566966:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495715456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.566973:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.566977:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575412.566980:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495715456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575413.526316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575413.526374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575414.550288:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575414.550358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575415.574329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575415.574375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575416.598338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575416.598371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575417.622301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575417.622346:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575417.686309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575417.686321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575417.686326:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575417.686355:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3634 00000100:00080000:0.0:1788575417.686366:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575417.686374:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575417.686412:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495715584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575417.686444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575417.686448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575417.686450:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575417.686455:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3634 00000100:00080000:0.0:1788575417.686460:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575417.686465:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575417.686476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575417.686493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495715712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575417.686506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495715584:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575417.686509:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575417.686517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575417.686519:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575417.686524:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3634 00000100:00080000:0.0:1788575417.686528:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575417.686533:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575417.686539:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575417.686553:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495715840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575417.686577:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575417.686578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575417.686584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575417.686586:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575417.686601:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3634 00000100:00080000:0.0:1788575417.686608:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575417.686613:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575417.686619:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575417.686631:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff93a2460d0380 x1875453495715584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575417.686651:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495715968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575417.686674:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495715584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.686691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.686695:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575417.686698:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495715584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575417.686717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575417.686753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575417.686767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff93a2460d1500 x1875453495715712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.686781:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495715712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.686790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.686793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575417.686797:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575417.686804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715840:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575417.686827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575417.686838:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff93a2460d2680 x1875453495715840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.686846:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495715840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.686852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.686855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575417.686857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495715840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575417.686867:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495715968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575417.686882:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575417.686889:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff93a2460d3480 x1875453495715968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.686896:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495715968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.686902:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.686904:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575417.686906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495715968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575418.646334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575418.646401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575419.670330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575419.670420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575420.694305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575420.694355:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575421.718295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575421.718385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575422.742290:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575422.742361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575422.806323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575422.806335:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575422.806340:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575422.806351:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3639 00000100:00080000:0.0:1788575422.806362:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575422.806370:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575422.806401:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495716096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575422.806425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575422.806428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575422.806441:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575422.806445:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3639 00000100:00080000:0.0:1788575422.806448:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575422.806452:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575422.806461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575422.806471:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495716224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575422.806477:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495716096:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575422.806482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575422.806485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575422.806487:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575422.806491:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3639 00000100:00080000:0.0:1788575422.806493:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575422.806495:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575422.806499:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575422.806508:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495716352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575422.806515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575422.806518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575422.806519:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575422.806522:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3639 00000100:00080000:0.0:1788575422.806531:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575422.806533:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575422.806537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575422.806539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575422.806554:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495716480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575422.806579:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff93a2460d2300 x1875453495716096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.806612:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495716096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.806621:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.806624:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575422.806626:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495716096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575422.806652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575422.806671:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575422.806680:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff93a2460d1f80 x1875453495716224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.806697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495716224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.806703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.806705:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575422.806707:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575422.806711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716352:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575422.806734:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575422.806741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff93a2460d2a00 x1875453495716352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.806750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495716352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.806759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.806763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575422.806766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716352:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575422.806773:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495716480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575422.806794:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575422.806806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff93a2460d1180 x1875453495716480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.806816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495716480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.806831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.806834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575422.806836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495716480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575423.766334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575423.766396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575424.790305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575424.790376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575425.814329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575425.814371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575426.838311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575426.838367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575427.862344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575427.862417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575427.926402:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575427.926413:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575427.926418:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575427.926428:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3644 00000100:00080000:3.0:1788575427.926439:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575427.926446:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575427.926492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495716608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575427.926514:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575427.926517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575427.926518:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575427.926521:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3644 00000100:00080000:3.0:1788575427.926523:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575427.926526:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575427.926535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575427.926565:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495716736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575427.926572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495716608:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575427.926576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575427.926579:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575427.926604:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575427.926610:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3644 00000100:00080000:3.0:1788575427.926615:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575427.926620:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575427.926626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575427.926641:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575427.926642:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495716864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575427.926652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575427.926655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575427.926656:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575427.926660:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3644 00000100:00080000:3.0:1788575427.926663:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575427.926666:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575427.926670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575427.926678:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495716992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575427.926698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff93a25064d180 x1875453495716608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575427.926715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495716608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575427.926724:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575427.926727:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575427.926729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495716608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575427.926755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575427.926778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575427.926791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff93a25064c000 x1875453495716736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575427.926803:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495716736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575427.926811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575427.926815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575427.926818:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575427.926839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716864:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575427.926861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575427.926869:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff93a25064f480 x1875453495716864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575427.926877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495716864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575427.926882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575427.926884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575427.926886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495716864:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575427.926890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495716992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575427.926900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575427.926905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff93a25064fb80 x1875453495716992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575427.926911:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495716992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575427.926916:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575427.926917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575427.926919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495716992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575428.886331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575428.886382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575429.910333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575429.910392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575430.934335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575430.934383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575431.958330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575431.958381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575432.982350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575432.982423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575433.046333:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575433.046344:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575433.046348:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575433.046368:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3649 00000100:00080000:3.0:1788575433.046400:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575433.046407:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575433.046448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495717120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575433.046473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575433.046476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575433.046477:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575433.046481:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3649 00000100:00080000:3.0:1788575433.046483:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575433.046485:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575433.046506:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575433.046517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495717248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575433.046524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575433.046527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575433.046528:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575433.046532:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3649 00000100:00080000:3.0:1788575433.046535:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575433.046540:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495717120:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575433.046540:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575433.046546:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575433.046556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495717376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575433.046568:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575433.046577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575433.046580:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575433.046597:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3649 00000100:00080000:3.0:1788575433.046602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575433.046607:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575433.046613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575433.046617:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575433.046630:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495717504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575433.046681:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff93a25064e680 x1875453495717120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.046709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495717120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.046720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.046725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575433.046729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495717120:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575433.046736:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575433.046756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575433.046767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff93a25064ea00 x1875453495717248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.046778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495717248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.046787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.046789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575433.046793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575433.046831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717376:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575433.046848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575433.046857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff93a25064d880 x1875453495717376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.046868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495717376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.046887:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.046890:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575433.046893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717376:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575433.046899:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495717504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575433.046914:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575433.046923:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff93a25064dc00 x1875453495717504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.046933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495717504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.046941:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.046944:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575433.046947:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495717504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575434.006287:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575434.006358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575435.030337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575435.030402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575436.054304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575436.054354:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575437.078339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575437.078396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575438.102335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575438.102395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575438.166313:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575438.166321:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575438.166325:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575438.166333:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3654 00000100:00080000:1.0:1788575438.166342:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575438.166348:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575438.166379:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495717632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575438.166401:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575438.166417:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575438.166419:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575438.166423:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3654 00000100:00080000:1.0:1788575438.166426:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575438.166430:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575438.166438:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495717632:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788575438.166439:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575438.166466:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495717760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575438.166475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575438.166477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575438.166479:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575438.166482:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3654 00000100:00080000:1.0:1788575438.166485:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575438.166487:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575438.166491:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575438.166499:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495717888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575438.166506:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575438.166511:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575438.166512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575438.166515:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575438.166519:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3654 00000100:00080000:1.0:1788575438.166521:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575438.166524:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575438.166533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575438.166542:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495718016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575438.166553:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff93a258da0700 x1875453495717632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575438.166574:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495717632/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575438.166586:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575438.166590:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575438.166595:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495717632:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575438.166635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575438.166668:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575438.166679:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff93a258da1c00 x1875453495717760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575438.166688:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495717760/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575438.166694:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575438.166696:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575438.166698:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575438.166702:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717888:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575438.166715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575438.166721:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff93a258da3800 x1875453495717888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575438.166728:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495717888/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575438.166733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575438.166735:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575438.166737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495717888:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575438.166740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495718016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575438.166750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575438.166755:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff93a258da1880 x1875453495718016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575438.166773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495718016/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575438.166779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575438.166780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575438.166782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495718016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575439.126318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575439.126363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575440.150356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575440.150462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575441.174381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575441.174466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575442.198361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575442.198431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575443.222361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575443.222437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575443.286380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575443.286393:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575443.286409:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575443.286423:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3659 00000100:00080000:1.0:1788575443.286436:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575443.286445:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575443.286490:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495718144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575443.286523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575443.286527:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575443.286529:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575443.286533:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3659 00000100:00080000:1.0:1788575443.286536:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575443.286540:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575443.286561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575443.286577:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495718272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575443.286586:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495718144:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575443.286591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575443.286596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575443.286598:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575443.286603:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3659 00000100:00080000:1.0:1788575443.286629:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575443.286634:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575443.286641:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575443.286652:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495718400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575443.286663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575443.286667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1788575443.286672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575443.286672:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575443.286680:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3659 00000100:00080000:1.0:1788575443.286684:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575443.286689:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575443.286693:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575443.286704:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495718528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575443.286731:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff93a258da3100 x1875453495718144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575443.286751:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495718144/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575443.286762:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575443.286766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575443.286770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495718144:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575443.286798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575443.286819:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575443.286831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff93a258da2a00 x1875453495718272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575443.286852:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495718272/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575443.286864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575443.286877:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575443.286881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575443.286887:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718400:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575443.286908:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575443.286919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff93a258da3480 x1875453495718400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575443.286929:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495718400/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575443.286937:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575443.286940:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575443.286943:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718400:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575443.286949:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495718528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575443.286966:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575443.286975:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff93a258da1180 x1875453495718528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575443.286986:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495718528/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575443.287004:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575443.287008:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575443.287011:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495718528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575444.246368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575444.246433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575445.270311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575445.270366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575446.294319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575446.294386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575447.318321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575447.318461:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575448.342291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575448.342332:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575448.406329:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575448.406338:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575448.406342:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575448.406377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3664 00000100:00080000:1.0:1788575448.406388:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575448.406399:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575448.406426:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495718656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575448.406447:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575448.406466:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575448.406467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575448.406470:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3664 00000100:00080000:1.0:1788575448.406473:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575448.406477:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575448.406485:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575448.406494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495718784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575448.406501:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495718656:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575448.406504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575448.406510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575448.406511:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575448.406514:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3664 00000100:00080000:1.0:1788575448.406516:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575448.406518:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575448.406521:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575448.406529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495718912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575448.406535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575448.406538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575448.406539:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575448.406541:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3664 00000100:00080000:1.0:1788575448.406554:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575448.406557:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788575448.406566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788575448.406568:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575448.406579:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495719040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575448.406608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff93a258da0380 x1875453495718656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575448.406636:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495718656/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575448.406645:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575448.406648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575448.406652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495718656:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575448.406674:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575448.406694:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575448.406707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff93a258da0000 x1875453495718784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575448.406719:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495718784/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575448.406734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575448.406738:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575448.406741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575448.406747:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718912:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575448.406768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575448.406778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff93a258da2300 x1875453495718912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575448.406787:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495718912/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575448.406794:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575448.406796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575448.406799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495718912:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575448.406804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495719040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575448.406818:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575448.406827:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff93a258da1180 x1875453495719040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575448.406836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495719040/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575448.406842:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575448.406845:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575448.406853:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495719040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575449.366327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575449.366377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575450.390328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575450.390395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575451.414319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575451.414371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575452.438291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575452.438300:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575453.462321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575453.462361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575453.526338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.526352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575453.526356:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.526364:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3669 00000100:00080000:0.0:1788575453.526372:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.526377:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575453.526409:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495719168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575453.526434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.526437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575453.526439:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.526444:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3669 00000100:00080000:0.0:1788575453.526448:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.526453:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575453.526462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575453.526468:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495719168:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575453.526479:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495719296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575453.526490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.526508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575453.526511:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.526515:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3669 00000100:00080000:0.0:1788575453.526520:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.526525:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575453.526529:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788575453.526533:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575453.526548:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495719424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575453.526559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.526563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1788575453.526565:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff93a2460d1180 x1875453495719168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575453.526566:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.526575:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3669 00000100:00080000:0.0:1788575453.526582:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575453.526586:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495719168/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575453.526587:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575453.526596:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788575453.526599:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.526603:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575453.526607:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495719168:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575453.526612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495719552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575453.526632:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575453.526656:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575453.526668:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff93a2460d2a00 x1875453495719296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.526676:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495719296/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.526681:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.526684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575453.526688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575453.526693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719424:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575453.526719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575453.526730:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff93a2460d1f80 x1875453495719424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.526737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495719424/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.526742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.526744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575453.526746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719424:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575453.526750:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495719552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575453.526762:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575453.526775:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff93a2460d2300 x1875453495719552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.526785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495719552/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.526790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.526792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575453.526794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495719552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575454.486322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575454.486381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575455.510374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575455.510439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575456.534327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575456.534385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575457.558332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575457.558829:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575458.582555:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575458.584418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575458.646336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.646345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575458.646349:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.646359:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3675 00000100:00080000:1.0:1788575458.646369:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.646391:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575458.646421:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495719680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.646449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.646453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575458.646454:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.646458:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3675 00000100:00080000:1.0:1788575458.646462:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.646466:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575458.646474:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575458.646490:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495719680:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575458.646490:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495719808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.646505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.646508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575458.646510:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.646514:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3675 00000100:00080000:1.0:1788575458.646518:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.646522:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575458.646536:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575458.646549:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495719936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.646563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.646567:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1788575458.646572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719808:192.168.202.102@tcp:38:kworker.0 00000100:00080000:1.0:1788575458.646576:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.646581:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3675 00000100:00080000:1.0:1788575458.646592:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.646598:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575458.646605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575458.646617:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719936:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575458.646620:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495720064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575458.646652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495720064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575459.606602:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575459.608402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575460.630571:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575460.631587:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575461.654865:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575461.655848:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575461.718550:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788575461.718607:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575461.718629:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575461.718678:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575461.718690:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575461.718700:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788575461.718734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff93a258da1880 x1875453495720064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575461.718763:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495720064/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575461.718776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575461.718781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575461.718786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495720064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575461.718795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff93a258da2a00 x1875453495719808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575461.718808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495719808/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575461.718816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575461.718820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575461.718824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575461.718831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff93a258da3100 x1875453495719936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575461.718843:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495719936/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575461.718872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575461.718876:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575461.718880:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495719936:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575461.718887:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff93a258da3480 x1875453495719680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575461.718901:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495719680/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575461.718909:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575461.718912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575461.718915:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495719680:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575462.678455:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575462.678471:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575463.702307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575463.702334:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575463.766417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.766428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575463.766435:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.766446:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3680 00000100:00080000:2.0:1788575463.766456:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.766463:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575463.766505:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495720192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.766554:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.766558:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.766560:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.766565:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3680 00000100:00080000:2.0:1788575463.766569:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.766573:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575463.766605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.766620:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495720320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.766631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.766634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.766636:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.766641:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3680 00000100:00080000:2.0:1788575463.766645:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.766649:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575463.766654:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.766664:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495720448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.766673:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.766677:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.766678:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.766682:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3680 00000100:00080000:2.0:1788575463.766685:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.766688:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575463.766693:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.766709:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495720576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575463.766831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495720192:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575463.766922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575463.766986:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff93a249409880 x1875453495720192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.767022:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495720192/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.767033:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.767038:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575463.767042:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495720192:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575463.767050:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575463.767087:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575463.767100:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff93a249408000 x1875453495720320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.767112:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495720320/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.767127:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.767131:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575463.767134:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575463.767145:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720448:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575463.767167:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575463.767179:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff93a24940aa00 x1875453495720448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.767190:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495720448/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.767199:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.767205:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575463.767208:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720448:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575463.767213:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495720576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575463.767289:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575463.767304:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff93a249409500 x1875453495720576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.767330:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495720576/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.767338:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.767342:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575463.767349:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495720576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575464.726310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575464.726327:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575465.750303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575465.750331:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788575466.774347:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575466.774347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575467.798308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:1.0:1788575467.798324:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575468.822319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575468.822351:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575468.886313:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575468.886323:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575468.886327:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575468.886336:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3685 00000100:00080000:3.0:1788575468.886347:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575468.886353:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575468.886380:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495720704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575468.886400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575468.886403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575468.886405:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575468.886409:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3685 00000100:00080000:3.0:1788575468.886412:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575468.886415:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575468.886425:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575468.886438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495720832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575468.886441:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495720704:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575468.886448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575468.886462:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575468.886464:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575468.886468:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3685 00000100:00080000:3.0:1788575468.886471:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575468.886474:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575468.886478:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575468.886487:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495720960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575468.886496:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575468.886496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575468.886499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575468.886500:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575468.886503:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3685 00000100:00080000:3.0:1788575468.886506:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575468.886508:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575468.886512:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575468.886519:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495721088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575468.886536:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff93a25064ed80 x1875453495720704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575468.886567:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495720704/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575468.886578:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575468.886582:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575468.886585:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495720704:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575468.886603:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575468.886620:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575468.886628:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff93a25064dc00 x1875453495720832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575468.886635:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495720832/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575468.886653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575468.886656:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575468.886658:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575468.886661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720960:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575468.886675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575468.886681:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff93a25064d880 x1875453495720960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575468.886687:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495720960/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575468.886691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575468.886693:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575468.886694:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495720960:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575468.886698:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495721088:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575468.886708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575468.886717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff93a25064ea00 x1875453495721088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575468.886727:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495721088/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575468.886733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575468.886736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575468.886744:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495721088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575469.846365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575469.846457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575470.870340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575470.870379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575471.894382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575471.894393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575472.918341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575472.918391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575473.942321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575473.942410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575474.006344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575474.006354:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575474.006381:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575474.006392:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3690 00000100:00080000:3.0:1788575474.006404:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575474.006412:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575474.006458:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495721216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575474.006497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575474.006501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575474.006503:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575474.006506:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3690 00000100:00080000:3.0:1788575474.006510:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575474.006513:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575474.006523:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575474.006536:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495721344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575474.006543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575474.006546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575474.006548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575474.006551:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3690 00000100:00080000:3.0:1788575474.006552:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575474.006555:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575474.006558:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575474.006562:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495721216:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575474.006567:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495721472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575474.006592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575474.006595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575474.006597:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575474.006600:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3690 00000100:00080000:3.0:1788575474.006603:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575474.006606:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575474.006610:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575474.006619:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495721600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575474.006651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575474.006751:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff93a25064e680 x1875453495721216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.006778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495721216/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.006816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.006822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575474.006827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495721216:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575474.006837:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721344:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575474.006882:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575474.006898:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff93a25064fb80 x1875453495721344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.006912:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495721344/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.006922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.006925:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575474.006929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575474.006961:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721472:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575474.006986:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575474.007008:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff93a25064f480 x1875453495721472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.007020:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495721472/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.007029:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.007033:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575474.007037:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721472:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575474.007044:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495721600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575474.007064:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575474.007075:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff93a25064c000 x1875453495721600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.007086:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495721600/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.007094:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.007097:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575474.007100:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495721600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575474.966350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575474.966400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575475.990379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575475.990442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575477.014359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575477.014456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575478.038371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575478.038423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575479.062365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575479.062415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575479.126361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575479.126372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575479.126376:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575479.126387:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3695 00000100:00080000:2.0:1788575479.126398:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575479.126405:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575479.126450:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495721728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575479.126481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575479.126484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575479.126485:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575479.126489:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3695 00000100:00080000:2.0:1788575479.126491:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575479.126494:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575479.126520:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575479.126533:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495721728:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575479.126534:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495721856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575479.126546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575479.126548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575479.126550:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575479.126553:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3695 00000100:00080000:2.0:1788575479.126555:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575479.126557:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575479.126561:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575479.126568:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495721984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575479.126575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575479.126578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575479.126579:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575479.126581:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3695 00000100:00080000:2.0:1788575479.126589:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575479.126592:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575479.126594:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575479.126602:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495722112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575479.126613:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575479.126664:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff93a24940b480 x1875453495721728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.126680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495721728/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.126695:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.126698:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575479.126701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495721728:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575479.126726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575479.126750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575479.126763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff93a24940bb80 x1875453495721856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.126775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495721856/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.126783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.126786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575479.126790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575479.126794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721984:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575479.126810:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575479.126817:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff93a24940b100 x1875453495721984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.126824:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495721984/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.126829:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.126869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575479.126872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495721984:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575479.126881:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495722112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575479.126897:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575479.126904:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff93a249409500 x1875453495722112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.126911:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495722112/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.126916:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.126918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575479.126920:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495722112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575480.086333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575480.086396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575481.110352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575481.110412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575482.134327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575482.134382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575483.158313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575483.158374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575484.182340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575484.182389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575484.246360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575484.246370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575484.246375:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575484.246386:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3700 00000100:00080000:0.0:1788575484.246396:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575484.246402:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575484.246438:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495722240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575484.246469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575484.246473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575484.246475:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575484.246483:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3700 00000100:00080000:0.0:1788575484.246487:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575484.246506:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575484.246516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495722240:192.168.202.102@tcp:250:kworker.0 00000008:00100000:0.0:1788575484.246517:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575484.246534:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495722368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575484.246546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575484.246549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575484.246551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575484.246556:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3700 00000100:00080000:0.0:1788575484.246560:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575484.246564:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575484.246569:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575484.246580:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495722496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575484.246589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575484.246592:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575484.246594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575484.246597:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575484.246602:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3700 00000100:00080000:0.0:1788575484.246607:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575484.246611:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575484.246615:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575484.246626:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495722624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575484.246639:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff93a2460d3480 x1875453495722240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575484.246659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495722240/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575484.246670:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575484.246674:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575484.246677:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495722240:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575484.246699:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495722368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575484.246720:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575484.246740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff93a2460d2680 x1875453495722368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575484.246752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495722368/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575484.246759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575484.246763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575484.246766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495722368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575484.246771:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495722496:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575484.246791:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575484.246802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff93a2460d1500 x1875453495722496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575484.246813:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495722496/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575484.246820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575484.246823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575484.246826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495722496:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575484.246832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495722624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575484.246850:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575484.246860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff93a2460d0380 x1875453495722624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575484.246871:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495722624/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575484.246884:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575484.246887:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575484.246890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495722624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575485.206303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575485.206373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575486.230324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575486.230393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575487.254303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575487.254390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575488.278365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575488.278421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575489.302377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575489.302451:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575489.366351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575489.366362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575489.366366:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575489.366375:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3705 00000100:00080000:1.0:1788575489.366386:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575489.366393:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575489.366429:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495722752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575489.366455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575489.366459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575489.366462:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575489.366466:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3705 00000100:00080000:1.0:1788575489.366469:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575489.366473:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575489.366482:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575489.366493:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495722880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575489.366497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495722752:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575489.366505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575489.366508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575489.366518:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575489.366523:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3705 00000100:00080000:1.0:1788575489.366526:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575489.366529:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575489.366534:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575489.366545:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495723008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575489.366554:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575489.366558:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575489.366560:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575489.366563:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3705 00000100:00080000:1.0:1788575489.366566:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575489.366569:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575489.366573:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575489.366575:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575489.366585:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495723136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575489.366619:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff93a258da3800 x1875453495722752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.366637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495722752/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.366648:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.366652:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575489.366656:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495722752:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575489.366678:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495722880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575489.366699:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575489.366712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff93a258da1c00 x1875453495722880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.366723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495722880/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.366731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.366734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575489.366738:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495722880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575489.366744:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723008:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575489.366763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575489.366774:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff93a258da0700 x1875453495723008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.366799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495723008/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.366806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.366810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575489.366813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723008:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575489.366818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495723136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575489.366839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575489.366849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff93a258da0e00 x1875453495723136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.366860:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495723136/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.366868:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.366871:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575489.366874:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495723136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575490.326370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575490.326437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575491.350422:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575491.350499:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575492.374327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575492.374387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575493.398385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575493.398452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575494.422342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575494.422402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575494.486309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.486319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575494.486323:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.486333:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3710 00000100:00080000:1.0:1788575494.486342:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.486350:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575494.486383:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495723264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.486428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.486431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575494.486433:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.486438:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3710 00000100:00080000:1.0:1788575494.486443:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.486448:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575494.486458:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575494.486468:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495723264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575494.486474:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495723392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.486485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.486489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575494.486491:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.486495:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3710 00000100:00080000:1.0:1788575494.486499:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.486504:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575494.486509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575494.486518:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575494.486522:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495723520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.486542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.486547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575494.486548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.486552:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3710 00000100:00100000:0.0:1788575494.486558:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff93a258da1f80 x1875453495723264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575494.486562:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.486569:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575494.486575:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575494.486587:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495723264/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575494.486590:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495723648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575494.486627:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.486632:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575494.486636:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495723264:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575494.486657:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575494.486678:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575494.486690:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff93a258da0a80 x1875453495723392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.486701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495723392/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.486719:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.486722:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575494.486725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575494.486731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723520:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575494.486756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575494.486766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff93a258da2680 x1875453495723520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.486776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495723520/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.486784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.486787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575494.486790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723520:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575494.486804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495723648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575494.486821:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575494.486830:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff93a258da0e00 x1875453495723648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.486840:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495723648/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.486845:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.486847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575494.486848:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495723648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575495.446362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575495.446408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575496.470338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575496.470401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575497.494330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575497.494419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575498.518331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575498.518437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575499.542357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575499.542431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575499.606366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.606383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575499.606388:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.606399:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3716 00000100:00080000:1.0:1788575499.606410:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.606417:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575499.606469:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495723776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575499.606496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.606499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575499.606501:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.606504:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3716 00000100:00080000:1.0:1788575499.606506:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.606509:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575499.606518:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575499.606528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495723904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575499.606539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575499.606547:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495723776:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575499.606563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575499.606566:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.606571:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3716 00000100:00080000:1.0:1788575499.606577:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.606582:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575499.606588:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575499.606602:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495724032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575499.606614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.606618:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575499.606620:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.606623:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3716 00000100:00080000:1.0:1788575499.606628:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575499.606633:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575499.606635:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575499.606642:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575499.606655:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495724160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575499.606684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff93a258da0700 x1875453495723776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.606701:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495723776/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.606710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.606713:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575499.606716:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495723776:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575499.606741:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575499.606763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575499.606776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff93a258da1c00 x1875453495723904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.606797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495723904/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.606805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.606809:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575499.606812:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495723904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575499.606818:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724032:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575499.606835:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575499.606842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff93a258da3800 x1875453495724032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.606849:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495724032/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.606854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.606857:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575499.606858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724032:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575499.606861:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495724160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575499.606871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575499.606876:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff93a258da3480 x1875453495724160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.606887:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495724160/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.606906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.606909:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575499.606913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495724160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575500.566350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575500.566409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575501.590363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575501.590431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575502.614343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575502.614407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575503.638321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575503.638413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575504.662310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575504.662341:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575504.726332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575504.726340:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575504.726343:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575504.726351:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3721 00000100:00080000:0.0:1788575504.726360:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575504.726365:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575504.726393:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495724288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575504.726413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575504.726416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575504.726418:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575504.726422:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3721 00000100:00080000:0.0:1788575504.726425:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575504.726429:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575504.726439:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575504.726451:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495724416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575504.726455:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495724288:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575504.726466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575504.726470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575504.726486:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575504.726491:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3721 00000100:00080000:0.0:1788575504.726495:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575504.726499:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575504.726505:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575504.726517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495724544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575504.726526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575504.726530:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575504.726531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575504.726534:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575504.726538:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3721 00000100:00080000:0.0:1788575504.726543:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575504.726547:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575504.726551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575504.726561:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495724672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575504.726584:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff93a2460d0a80 x1875453495724288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.726604:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495724288/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.726628:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.726632:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575504.726636:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495724288:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575504.726660:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575504.726684:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575504.726696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff93a2460d0000 x1875453495724416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.726708:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495724416/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.726716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.726719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575504.726722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575504.726728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724544:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575504.726747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575504.726756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff93a2460d3b80 x1875453495724544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.726766:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495724544/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.726774:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.726777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575504.726780:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724544:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575504.726791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495724672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575504.726807:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575504.726816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff93a2460d0700 x1875453495724672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.726827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495724672/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.726834:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.726837:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575504.726840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495724672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575505.686334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575505.686373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575506.710340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575506.710440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575507.734319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575507.734376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575508.758344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575508.758403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575509.782334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575509.782387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575509.846329:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575509.846341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575509.846345:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575509.846352:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3726 00000100:00080000:0.0:1788575509.846360:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575509.846365:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575509.846399:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495724800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575509.846442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575509.846446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575509.846447:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575509.846452:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3726 00000100:00080000:0.0:1788575509.846457:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575509.846461:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575509.846470:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575509.846482:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495724928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575509.846483:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495724800:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575509.846491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575509.846494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575509.846496:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575509.846499:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3726 00000100:00080000:0.0:1788575509.846502:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575509.846506:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575509.846509:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575509.846517:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495725056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575509.846524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575509.846527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575509.846528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575509.846531:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3726 00000100:00080000:0.0:1788575509.846539:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575509.846543:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575509.846544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788575509.846547:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575509.846555:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495725184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575509.846576:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff93a2460d0700 x1875453495724800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.846591:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495724800/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.846600:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.846604:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575509.846607:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495724800:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575509.846627:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575509.846643:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575509.846651:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff93a2460d3b80 x1875453495724928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.846666:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495724928/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.846674:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.846677:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575509.846680:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495724928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575509.846686:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725056:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575509.846708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575509.846734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff93a2460d0000 x1875453495725056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.846745:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495725056/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.846752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.846755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575509.846758:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725056:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575509.846763:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495725184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575509.846783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575509.846794:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff93a2460d0a80 x1875453495725184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.846805:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495725184/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.846823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.846826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575509.846829:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495725184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575510.806336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575510.806382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575511.830313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575511.830392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575512.854354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575512.854472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575513.878300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575513.878405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575514.902378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575514.902447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575514.966349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575514.966362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575514.966367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575514.966380:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3731 00000100:00080000:1.0:1788575514.966392:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575514.966400:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575514.966444:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495725312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575514.966477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575514.966481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575514.966483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575514.966488:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3731 00000100:00080000:1.0:1788575514.966492:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575514.966498:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575514.966510:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575514.966513:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495725312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575514.966526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495725440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575514.966538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575514.966542:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575514.966563:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575514.966569:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3731 00000100:00080000:1.0:1788575514.966573:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575514.966579:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575514.966580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575514.966586:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575514.966599:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495725568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575514.966610:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575514.966614:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575514.966616:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575514.966620:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3731 00000100:00080000:1.0:1788575514.966624:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575514.966628:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575514.966633:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575514.966641:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff93a258da3100 x1875453495725312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575514.966643:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495725696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575514.966678:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495725312/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575514.966691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575514.966695:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575514.966699:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495725312:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575514.966726:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575514.966748:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575514.966760:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff93a258da2a00 x1875453495725440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575514.966772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495725440/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575514.966790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575514.966793:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575514.966804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575514.966811:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725568:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575514.966836:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575514.966848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff93a258da1880 x1875453495725568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575514.966874:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495725568/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575514.966882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575514.966886:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575514.966889:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725568:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575514.966895:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495725696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575514.966916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575514.966927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff93a258da1180 x1875453495725696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575514.966939:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495725696/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575514.966947:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575514.966950:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575514.966954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495725696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575515.926356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575515.926444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575516.950359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575516.950468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575517.974355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575517.974422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575518.998341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575518.998391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575520.022316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575520.022364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575520.086331:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575520.086342:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575520.086362:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575520.086373:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3736 00000100:00080000:0.0:1788575520.086382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575520.086389:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575520.086418:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495725824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575520.086440:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575520.086442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575520.086443:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575520.086446:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3736 00000100:00080000:0.0:1788575520.086449:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575520.086454:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575520.086463:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575520.086467:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495725824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575520.086478:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495725952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575520.086489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575520.086493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575520.086495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575520.086500:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3736 00000100:00080000:0.0:1788575520.086511:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575520.086514:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575520.086518:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788575520.086519:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575520.086530:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495726080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575520.086546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575520.086549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575520.086550:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575520.086554:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff93a2460d0380 x1875453495725824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575520.086557:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3736 00000100:00080000:0.0:1788575520.086562:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575520.086567:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575520.086571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575520.086575:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495725824/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575520.086585:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575520.086586:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495726208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575520.086599:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575520.086614:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495725824:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575520.086636:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575520.086652:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575520.086661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff93a2460d1500 x1875453495725952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575520.086668:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495725952/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575520.086679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575520.086681:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575520.086683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495725952:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575520.086687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726080:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575520.086698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575520.086704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff93a2460d2680 x1875453495726080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575520.086710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495726080/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575520.086714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575520.086716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575520.086718:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726080:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575520.086721:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495726208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575520.086736:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575520.086742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff93a2460d3480 x1875453495726208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575520.086749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495726208/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575520.086753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575520.086760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575520.086762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495726208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575521.046331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575521.046427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575522.070293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575522.070317:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575523.094329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575523.094396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575524.118311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575524.118557:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575525.142569:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575525.142951:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575525.206345:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575525.206353:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575525.206356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575525.206364:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3741 00000100:00080000:3.0:1788575525.206372:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575525.206378:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575525.206406:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495726336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575525.206426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575525.206428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575525.206433:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575525.206435:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3741 00000100:00080000:3.0:1788575525.206437:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575525.206440:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575525.206447:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575525.206455:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495726464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575525.206462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575525.206465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575525.206467:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495726336:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575525.206467:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575525.206474:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3741 00000100:00080000:3.0:1788575525.206476:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575525.206478:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575525.206481:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575525.206502:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495726592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575525.206508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575525.206511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575525.206512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575525.206514:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3741 00000100:00080000:3.0:1788575525.206516:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575525.206519:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575525.206521:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575525.206527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495726720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575525.206554:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575525.206581:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575525.206598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495726720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575526.166590:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:3.0:1788575526.167497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788575527.190584:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575527.190867:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575528.214328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575528.214575:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575528.278472:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788575528.278512:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575528.278528:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575528.278569:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575528.278578:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575528.278592:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575528.278616:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff93a25064d500 x1875453495726464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575528.278641:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495726464/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575528.278654:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575528.278660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575528.278664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575528.278672:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff93a25064f100 x1875453495726592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575528.278699:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495726592/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575528.278707:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575528.278710:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575528.278714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726592:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575528.278721:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff93a25064ce00 x1875453495726720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575528.278732:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495726720/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575528.278740:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575528.278743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575528.278747:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495726720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575528.278753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff93a25064d180 x1875453495726336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575528.278765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495726336/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575528.278773:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575528.278776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575528.278779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495726336:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575529.239058:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575529.239185:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575530.262354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575530.262485:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575530.326356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.326369:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575530.326374:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.326386:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3746 00000100:00080000:0.0:1788575530.326398:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.326406:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575530.326450:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495726848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.326483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.326489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575530.326490:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.326495:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3746 00000100:00080000:0.0:1788575530.326499:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.326504:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575530.326535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575530.326537:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495726848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575530.326557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495726976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.326569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.326572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575530.326574:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.326579:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3746 00000100:00080000:0.0:1788575530.326583:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.326588:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575530.326594:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575530.326605:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495727104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.326615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.326619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575530.326620:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.326625:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3746 00000100:00080000:0.0:1788575530.326629:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.326633:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575530.326638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575530.326644:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575530.326653:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495727232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575530.326714:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff93a2460d2300 x1875453495726848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.326737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495726848/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.326758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.326763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575530.326768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495726848:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575530.326811:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575530.326837:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575530.326852:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff93a2460d1f80 x1875453495726976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.326864:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495726976/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.326872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.326875:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575530.326878:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495726976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575530.326884:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495727104:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575530.326906:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575530.326918:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff93a2460d2a00 x1875453495727104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.326945:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495727104/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.326953:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.326956:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575530.326960:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495727104:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575530.326966:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495727232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575530.326990:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575530.327000:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff93a2460d1180 x1875453495727232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.327020:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495727232/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.327029:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.327032:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575530.327036:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495727232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575531.286293:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575531.286443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575532.310365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575532.310430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575533.334300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575533.334352:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575534.358327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575534.358374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575535.382364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575535.382445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575535.446350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575535.446363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575535.446368:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575535.446381:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3751 00000100:00080000:2.0:1788575535.446424:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575535.446433:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575535.446481:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495727360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575535.446508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575535.446510:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575535.446512:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575535.446515:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3751 00000100:00080000:2.0:1788575535.446517:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575535.446520:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575535.446530:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575535.446538:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495727488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575535.446545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575535.446547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575535.446549:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575535.446559:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3751 00000100:00080000:2.0:1788575535.446562:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575535.446565:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575535.446571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575535.446587:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495727616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575535.446590:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495727360:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575535.446613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575535.446627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575535.446630:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575535.446637:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3751 00000100:00080000:2.0:1788575535.446642:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575535.446646:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575535.446653:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575535.446667:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495727744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575535.446686:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575535.446733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff93a24940aa00 x1875453495727360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.446752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495727360/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.446781:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.446785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575535.446789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495727360:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575535.446796:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495727488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575535.446832:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575535.446843:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff93a249408000 x1875453495727488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.446854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495727488/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.446861:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.446864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575535.446867:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495727488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575535.446897:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495727616:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575535.446920:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575535.446931:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff93a249409880 x1875453495727616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.446942:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495727616/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.446949:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.446952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575535.446955:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495727616:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575535.446967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495727744:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575535.446989:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575535.446999:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff93a249408700 x1875453495727744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.447010:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495727744/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.447018:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.447021:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575535.447025:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495727744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575536.406329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575536.406384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575537.430304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575537.430385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575538.454310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575538.454363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575539.478365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575539.478416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575540.502360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575540.502421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575540.566401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.566416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575540.566443:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.566460:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3757 00000100:00080000:0.0:1788575540.566474:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.566483:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575540.566527:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495727872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.566560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.566564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575540.566566:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.566571:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3757 00000100:00080000:0.0:1788575540.566575:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.566579:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575540.566591:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575540.566606:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495728000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.566618:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.566621:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575540.566624:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.566633:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3757 00000100:00100000:1.0:1788575540.566637:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495727872:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575540.566638:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.566650:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575540.566656:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575540.566669:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495728128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.566693:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.566698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575540.566700:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575540.566703:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575540.566713:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3757 00000100:00080000:0.0:1788575540.566720:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.566726:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575540.566733:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575540.566746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff93a2460d1180 x1875453495727872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575540.566754:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495728256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575540.566767:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495727872/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.566778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.566781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575540.566784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495727872:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575540.566807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728000:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575540.566822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575540.566829:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff93a2460d2a00 x1875453495728000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.566848:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495728000/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.566854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.566856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575540.566857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575540.566861:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728128:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575540.566874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575540.566880:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff93a2460d1f80 x1875453495728128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.566886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495728128/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.566892:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.566894:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575540.566897:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728128:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575540.566902:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495728256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575540.566919:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575540.566929:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff93a2460d2300 x1875453495728256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.566940:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495728256/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.566956:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.566959:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575540.566962:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495728256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575541.526329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575541.526373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575542.550391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575542.550395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575543.574320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575543.574376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575544.598340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575544.598393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575545.622319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575545.622400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575545.686383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.686394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575545.686398:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.686408:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3762 00000100:00080000:2.0:1788575545.686420:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.686426:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575545.686477:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495728384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.686527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.686530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575545.686531:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.686534:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3762 00000100:00080000:2.0:1788575545.686537:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.686540:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575545.686549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575545.686560:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495728512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.686566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.686569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575545.686570:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.686573:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3762 00000100:00080000:2.0:1788575545.686575:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575545.686578:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495728384:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575545.686579:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575545.686589:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575545.686600:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495728640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.686607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.686611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575545.686613:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.686617:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3762 00000100:00080000:2.0:1788575545.686626:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.686631:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575545.686637:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575545.686649:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495728768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575545.686672:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575545.686722:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff93a249408a80 x1875453495728384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.686738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495728384/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.686747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.686750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575545.686753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495728384:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575545.686775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575545.686798:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575545.686811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff93a249409180 x1875453495728512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.686823:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495728512/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.686831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.686834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575545.686837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575545.686847:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728640:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575545.686864:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575545.686872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff93a24940ad80 x1875453495728640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.686879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495728640/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.686884:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.686886:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575545.686888:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495728640:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575545.686891:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495728768:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575545.686905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575545.686914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff93a24940b800 x1875453495728768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.686925:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495728768/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.686933:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.686936:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575545.686939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495728768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575546.646342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575546.646404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575547.670308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575547.670395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575548.694307:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575548.694392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575549.718345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575549.718399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575550.742304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575550.742358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575550.806357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575550.806367:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575550.806371:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575550.806399:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3767 00000100:00080000:1.0:1788575550.806410:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575550.806417:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575550.806448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495728896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575550.806476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575550.806479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575550.806481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575550.806485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3767 00000100:00080000:1.0:1788575550.806488:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575550.806492:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575550.806500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575550.806511:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495729024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575550.806521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575550.806526:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495728896:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575550.806526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575550.806540:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575550.806548:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3767 00000100:00080000:1.0:1788575550.806552:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575550.806555:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575550.806560:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575550.806571:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495729152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575550.806590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575550.806593:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575550.806595:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575550.806598:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3767 00000100:00080000:1.0:1788575550.806602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575550.806606:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575550.806610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575550.806615:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575550.806624:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495729280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575550.806657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff93a258da2300 x1875453495728896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575550.806670:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495728896/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575550.806677:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575550.806681:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575550.806684:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495728896:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575550.806708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575550.806730:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575550.806740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff93a258da0000 x1875453495729024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575550.806758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495729024/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575550.806766:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575550.806769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575550.806772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575550.806778:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729152:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575550.806800:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575550.806811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff93a258da0380 x1875453495729152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575550.806822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495729152/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575550.806831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575550.806834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575550.806836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729152:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575550.806842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495729280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575550.806861:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575550.806872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff93a258da1180 x1875453495729280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575550.806883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495729280/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575550.806901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575550.806905:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575550.806909:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495729280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575551.766305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575551.766358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575552.790304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575552.790364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575553.814335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575553.814389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575554.838366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575554.838420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575555.862384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575555.862492:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575555.926414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.926425:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575555.926430:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.926440:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3772 00000100:00080000:2.0:1788575555.926452:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.926460:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575555.926503:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495729408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575555.926530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.926533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575555.926535:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.926555:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3772 00000100:00080000:2.0:1788575555.926559:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.926562:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575555.926570:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495729408:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788575555.926573:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575555.926593:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495729536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575555.926601:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.926604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575555.926606:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.926624:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3772 00000100:00080000:2.0:1788575555.926628:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.926631:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575555.926637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575555.926649:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495729664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575555.926661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.926664:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575555.926666:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.926672:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3772 00000800:00000100:1.0:1788575555.926673:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575555.926679:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.926685:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575555.926691:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575555.926704:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495729792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575555.926723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff93a24940a680 x1875453495729408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.926739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495729408/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.926748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.926751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575555.926754:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495729408:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575555.926777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575555.926799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575555.926810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff93a249409c00 x1875453495729536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.926827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495729536/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.926833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.926835:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575555.926837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729536:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575555.926841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729664:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575555.926856:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575555.926862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff93a249408e00 x1875453495729664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.926869:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495729664/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.926874:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.926875:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575555.926877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495729664:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575555.926888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495729792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575555.926899:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575555.926905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff93a24940b800 x1875453495729792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.926912:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495729792/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.926925:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.926927:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575555.926929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495729792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575556.886318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575556.886366:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575557.910336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575557.910387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575558.934347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575558.934402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575559.958330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575559.958382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575560.982318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575560.982361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575561.046305:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575561.046316:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575561.046320:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575561.046330:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3777 00000100:00080000:2.0:1788575561.046340:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575561.046347:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575561.046394:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495729920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575561.046426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575561.046430:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575561.046432:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575561.046436:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3777 00000100:00080000:2.0:1788575561.046440:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575561.046443:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575561.046452:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575561.046463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495730048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575561.046472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575561.046475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575561.046489:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575561.046493:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3777 00000100:00080000:2.0:1788575561.046497:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575561.046501:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575561.046507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575561.046518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495730176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575561.046528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575561.046531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575561.046533:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575561.046536:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3777 00000100:00080000:2.0:1788575561.046539:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575561.046543:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575561.046547:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575561.046557:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495730304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575561.046644:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495729920:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788575561.046706:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575561.046744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff93a24940ad80 x1875453495729920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.046757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495729920/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.046763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.046766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575561.046770:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495729920:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575561.046777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575561.046799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575561.046811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff93a249409180 x1875453495730048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.046822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495730048/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.046827:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.046829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575561.046831:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575561.046841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730176:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575561.046856:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575561.046862:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff93a249408a80 x1875453495730176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.046868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495730176/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.046873:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.046875:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575561.046877:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730176:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575561.046880:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495730304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575561.046890:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575561.046895:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff93a249408700 x1875453495730304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.046900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495730304/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.046906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.046909:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575561.046912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495730304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575562.006325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575562.006367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575563.030298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575563.030371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575564.054325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575564.054372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575565.078326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575565.078390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575566.102316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575566.102395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575566.166335:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.166343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575566.166346:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.166359:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3782 00000100:00080000:0.0:1788575566.166367:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.166375:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575566.166405:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495730432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.166428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.166431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575566.166432:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.166435:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3782 00000100:00080000:0.0:1788575566.166437:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.166440:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575566.166448:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575566.166461:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495730560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.166470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.166472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575566.166489:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575566.166497:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495730432:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575566.166498:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3782 00000100:00080000:0.0:1788575566.166506:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.166514:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575566.166519:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575566.166534:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495730688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.166544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.166548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575566.166549:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.166553:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3782 00000100:00080000:0.0:1788575566.166578:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.166583:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575566.166587:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575566.166600:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495730816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575566.166605:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575566.166669:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff93a2460d3480 x1875453495730432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.166703:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495730432/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.166714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.166719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575566.166723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495730432:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575566.166731:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575566.166756:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575566.166782:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff93a2460d2680 x1875453495730560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.166796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495730560/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.166805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.166808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575566.166811:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575566.166840:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730688:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575566.166862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575566.166874:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff93a2460d1500 x1875453495730688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.166886:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495730688/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.166906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.166910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575566.166913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495730688:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575566.166920:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495730816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575566.166941:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575566.166952:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff93a2460d0380 x1875453495730816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.166964:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495730816/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.166971:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.166975:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575566.166978:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495730816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575567.126330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575567.126386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575568.150306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575568.150364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575569.174324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575569.174378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575570.198280:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575570.198343:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575571.222326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575571.222377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575571.286317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575571.286325:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575571.286328:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575571.286336:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3787 00000100:00080000:2.0:1788575571.286344:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575571.286350:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575571.286378:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495730944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575571.286396:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575571.286398:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575571.286399:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575571.286402:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3787 00000100:00080000:2.0:1788575571.286404:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575571.286406:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575571.286413:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575571.286421:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495731072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575571.286427:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575571.286429:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575571.286430:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575571.286434:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3787 00000100:00100000:1.0:1788575571.286436:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495730944:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575571.286436:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575571.286441:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575571.286446:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575571.286467:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495731200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575571.286474:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575571.286476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575571.286477:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575571.286479:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3787 00000100:00080000:2.0:1788575571.286481:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575571.286483:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575571.286486:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575571.286493:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495731328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575571.286498:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575571.286538:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff93a249409880 x1875453495730944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575571.286552:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495730944/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575571.286558:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575571.286561:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575571.286563:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495730944:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575571.286585:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575571.286600:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575571.286608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff93a249408000 x1875453495731072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575571.286615:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495731072/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575571.286619:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575571.286621:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575571.286623:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575571.286626:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731200:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575571.286635:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575571.286640:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff93a24940aa00 x1875453495731200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575571.286646:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495731200/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575571.286651:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575571.286652:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575571.286654:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731200:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575571.286657:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495731328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575571.286666:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575571.286671:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff93a249409500 x1875453495731328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575571.286681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495731328/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575571.286686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575571.286687:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575571.286689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495731328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575572.246309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575572.246371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575573.270331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575573.270398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575574.294355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575574.294409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575575.318326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575575.318369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575576.342332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575576.342392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575576.406325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575576.406334:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575576.406337:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575576.406346:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3792 00000100:00080000:2.0:1788575576.406356:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575576.406361:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575576.406391:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495731456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575576.406410:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575576.406413:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575576.406425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575576.406428:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3792 00000100:00080000:2.0:1788575576.406431:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575576.406434:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575576.406442:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575576.406454:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495731584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575576.406455:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495731456:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575576.406466:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575576.406469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575576.406470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575576.406473:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3792 00000100:00080000:2.0:1788575576.406475:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575576.406477:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575576.406480:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575576.406487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495731712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575576.406494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575576.406496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575576.406497:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575576.406499:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3792 00000100:00080000:2.0:1788575576.406501:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575576.406503:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575576.406506:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575576.406507:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575576.406521:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495731840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575576.406543:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff93a24940b100 x1875453495731456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.406561:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495731456/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.406572:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.406575:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575576.406579:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495731456:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575576.406598:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575576.406614:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575576.406622:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff93a24940bb80 x1875453495731584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.406629:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495731584/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.406643:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.406645:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575576.406647:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575576.406651:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731712:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575576.406663:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575576.406669:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff93a24940b480 x1875453495731712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.406676:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495731712/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.406680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.406682:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575576.406683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495731712:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575576.406687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495731840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575576.406696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575576.406701:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff93a249409500 x1875453495731840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.406707:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495731840/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.406711:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.406713:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575576.406718:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495731840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575577.366368:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575577.366425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575578.390345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575578.390404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575579.414335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575579.414386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575580.438322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575580.438377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575581.462400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575581.462444:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:1.0:1788575581.526367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575581.526378:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575581.526382:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575581.526392:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3797 00000100:00080000:1.0:1788575581.526403:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575581.526409:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575581.526450:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495731968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575581.526481:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575581.526485:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575581.526487:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575581.526491:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3797 00000100:00080000:1.0:1788575581.526495:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575581.526499:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575581.526508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575581.526522:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495732096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575581.526527:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495731968:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575581.526537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575581.526542:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575581.526543:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575581.526548:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3797 00000100:00080000:1.0:1788575581.526551:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575581.526570:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575581.526576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575581.526594:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575581.526599:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495732224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575581.526610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575581.526614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575581.526616:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575581.526619:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3797 00000100:00080000:1.0:1788575581.526624:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575581.526628:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575581.526633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575581.526635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff93a258da1880 x1875453495731968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.526646:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495732352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575581.526667:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495731968/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575581.526675:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575581.526678:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575581.526681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495731968:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575581.526703:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575581.526725:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575581.526744:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff93a258da2a00 x1875453495732096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575581.526752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495732096/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575581.526758:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575581.526759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575581.526762:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575581.526765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732224:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575581.526779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575581.526788:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff93a258da3100 x1875453495732224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575581.526798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495732224/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575581.526817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575581.526820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575581.526823:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732224:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575581.526829:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495732352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575581.526848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575581.526859:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff93a258da3480 x1875453495732352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575581.526870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495732352/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575581.526878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575581.526881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575581.526883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495732352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575582.486318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575582.486370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575583.510356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575583.510433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575584.534309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575584.534398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575585.558327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575585.558386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575586.582354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575586.582409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575586.646370:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.646380:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575586.646384:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.646395:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3803 00000100:00080000:2.0:1788575586.646405:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.646411:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575586.646453:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495732480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575586.646476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.646480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575586.646481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.646486:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3803 00000100:00080000:2.0:1788575586.646490:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.646495:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575586.646508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575586.646524:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495732480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575586.646525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495732608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575586.646539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.646543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575586.646545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.646550:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3803 00000100:00080000:2.0:1788575586.646554:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.646576:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575586.646583:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575586.646596:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575586.646607:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495732736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575586.646619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.646623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575586.646625:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.646629:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3803 00000100:00080000:2.0:1788575586.646638:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.646643:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575586.646648:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575586.646660:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff93a24940aa00 x1875453495732480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575586.646663:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495732864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575586.646685:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495732480/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575586.646700:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575586.646703:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575586.646707:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495732480:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575586.646733:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575586.646757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575586.646770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff93a249408000 x1875453495732608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575586.646780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495732608/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575586.646786:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575586.646788:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575586.646790:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732608:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575586.646794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732736:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575586.646809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575586.646816:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff93a249409880 x1875453495732736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575586.646822:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495732736/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575586.646827:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575586.646829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575586.646830:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495732736:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575586.646839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495732864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575586.646851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575586.646857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff93a249408700 x1875453495732864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575586.646864:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495732864/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575586.646869:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575586.646870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575586.646872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495732864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575587.606385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575587.606454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575588.630481:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575588.630582:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575589.654283:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575589.654359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575590.678410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575590.678528:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575591.702335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575591.702701:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575591.766354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575591.766364:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575591.766368:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575591.766377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3808 00000100:00080000:3.0:1788575591.766386:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575591.766393:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575591.766426:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495732992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575591.766460:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575591.766464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575591.766465:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575591.766469:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3808 00000100:00080000:3.0:1788575591.766473:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575591.766491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575591.766501:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575591.766512:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495732992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575591.766515:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495733120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575591.766528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575591.766532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575591.766534:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575591.766538:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3808 00000100:00080000:3.0:1788575591.766542:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575591.766546:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575591.766551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575591.766561:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495733248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575591.766570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575591.766573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575591.766575:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575591.766578:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3808 00000100:00080000:3.0:1788575591.766582:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575591.766586:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575591.766591:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575591.766601:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495733376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575591.766621:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575591.766650:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733248:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575591.766670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495733376:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788575592.726684:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575592.726804:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575593.750440:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575593.750555:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575594.774760:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575594.774845:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:1.0:1788575594.838440:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788575594.838475:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575594.838487:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575594.838516:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575594.838523:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575594.838528:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788575594.838566:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575594] req@ffff93a25064c700 x1875453495733120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575594.838606:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495733120/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575594.838617:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575594.838621:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575594.838625:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575594.838631:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575594] req@ffff93a25064ca80 x1875453495733248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575594.838640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495733248/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575594.838646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575594.838649:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575594.838652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733248:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575594.838658:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575594] req@ffff93a25064c380 x1875453495733376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575594.838668:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495733376/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575594.838674:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575594.838678:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575594.838680:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495733376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575594.838685:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575594] req@ffff93a25064e300 x1875453495732992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575594.838702:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495732992/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575594.838708:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575594.838711:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575594.838714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495732992:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575595.798723:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575595.799760:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575596.822350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575596.822426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575596.886358:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575596.886368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575596.886373:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575596.886384:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3813 00000100:00080000:3.0:1788575596.886394:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575596.886402:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575596.886439:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495733504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575596.886465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575596.886468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575596.886469:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575596.886472:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3813 00000100:00080000:3.0:1788575596.886474:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575596.886476:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575596.886485:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575596.886495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495733632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575596.886501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575596.886504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575596.886506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575596.886509:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3813 00000100:00080000:3.0:1788575596.886511:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575596.886513:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575596.886525:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495733504:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788575596.886534:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575596.886543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495733760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575596.886552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575596.886554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575596.886556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575596.886558:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3813 00000100:00080000:3.0:1788575596.886561:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575596.886563:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575596.886566:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575596.886574:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495733888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575596.886610:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575596.886670:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff93a25064df80 x1875453495733504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575596.886690:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495733504/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575596.886703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575596.886707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575596.886712:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495733504:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575596.886730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575596.886757:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575596.886770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff93a25064e300 x1875453495733632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575596.886780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495733632/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575596.886801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575596.886805:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575596.886808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575596.886841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733760:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575596.886862:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575596.886872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff93a25064c380 x1875453495733760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575596.886888:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495733760/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575596.886895:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575596.886898:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575596.886901:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495733760:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575596.886906:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495733888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575596.886922:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575596.886930:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff93a25064ca80 x1875453495733888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575596.886941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495733888/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575596.886949:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575596.886952:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575596.886961:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495733888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575597.846354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575597.846392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575598.870364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575598.870430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575599.894346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575599.894417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575600.918380:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575600.918434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575601.942334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575601.942442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575602.006362:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575602.006388:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575602.006393:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575602.006406:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3818 00000100:00080000:1.0:1788575602.006419:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575602.006449:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575602.006489:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495734016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575602.006517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575602.006522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575602.006523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575602.006528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3818 00000100:00080000:1.0:1788575602.006533:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575602.006538:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575602.006550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575602.006553:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495734016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575602.006566:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495734144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575602.006578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575602.006582:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575602.006584:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575602.006589:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3818 00000100:00080000:1.0:1788575602.006593:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575602.006597:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575602.006603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575602.006614:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495734272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575602.006631:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575602.006640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575602.006645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575602.006652:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575602.006658:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3818 00000100:00080000:1.0:1788575602.006663:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575602.006667:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575602.006673:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575602.006686:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495734400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575602.006690:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff93a258da3800 x1875453495734016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575602.006722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495734016/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575602.006737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575602.006744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575602.006749:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495734016:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575602.006776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575602.006793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575602.006802:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff93a258da1c00 x1875453495734144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575602.006810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495734144/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575602.006817:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575602.006820:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575602.006828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575602.006832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734272:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575602.006852:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575602.006863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff93a258da0700 x1875453495734272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575602.006873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495734272/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575602.006879:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575602.006882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575602.006885:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734272:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575602.006890:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495734400:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575602.006908:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575602.006917:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff93a258da0e00 x1875453495734400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575602.006928:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495734400/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575602.006935:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575602.006938:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575602.006941:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495734400:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788575602.966493:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575602.966547:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575603.990379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575603.990457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575605.014366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575605.014440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575606.038477:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575606.038545:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575607.062333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575607.062395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575607.126345:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575607.126355:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575607.126359:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575607.126367:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3823 00000100:00080000:3.0:1788575607.126375:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575607.126379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575607.126407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495734528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575607.126426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575607.126429:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575607.126430:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575607.126433:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3823 00000100:00080000:3.0:1788575607.126447:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575607.126450:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575607.126457:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575607.126465:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495734656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575607.126472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575607.126474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575607.126475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575607.126478:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3823 00000100:00080000:3.0:1788575607.126479:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575607.126481:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575607.126483:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575607.126490:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495734528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575607.126494:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495734784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575607.126502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575607.126504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575607.126506:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575607.126508:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3823 00000100:00080000:3.0:1788575607.126510:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575607.126512:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575607.126514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575607.126521:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495734912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575607.126552:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575607.126587:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff93a25064c700 x1875453495734528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575607.126604:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495734528/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575607.126613:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575607.126617:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575607.126620:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495734528:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575607.126627:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734656:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575607.126647:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575607.126658:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff93a25064d180 x1875453495734656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575607.126669:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495734656/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575607.126677:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575607.126680:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575607.126683:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575607.126705:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734784:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575607.126732:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575607.126742:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff93a25064ce00 x1875453495734784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575607.126752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495734784/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575607.126760:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575607.126763:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575607.126766:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495734784:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575607.126772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495734912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575607.126787:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575607.126796:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff93a25064f100 x1875453495734912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575607.126806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495734912/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575607.126819:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575607.126822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575607.126825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495734912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575608.086372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575608.086563:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575609.110414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575609.110466:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575610.134340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575610.134415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575611.158332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575611.158399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575612.182316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575612.182390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575612.246391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575612.246406:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575612.246411:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575612.246423:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3828 00000100:00080000:1.0:1788575612.246434:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575612.246441:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575612.246486:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495735040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575612.246510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575612.246512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575612.246514:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575612.246517:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3828 00000100:00080000:1.0:1788575612.246519:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575612.246521:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575612.246530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575612.246538:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495735168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575612.246567:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495735040:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575612.246568:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575612.246571:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575612.246573:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575612.246577:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3828 00000100:00080000:1.0:1788575612.246581:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575612.246585:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575612.246592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575612.246601:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495735296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575612.246608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575612.246611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575612.246612:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575612.246614:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3828 00000100:00080000:1.0:1788575612.246616:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575612.246618:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575612.246622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575612.246630:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495735424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575612.246650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575612.246708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff93a258da2680 x1875453495735040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575612.246726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495735040/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575612.246734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575612.246738:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575612.246741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495735040:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575612.246766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575612.246782:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575612.246791:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff93a258da0a80 x1875453495735168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575612.246799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495735168/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575612.246807:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575612.246810:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575612.246813:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575612.246819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735296:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575612.246839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575612.246849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff93a258da1f80 x1875453495735296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575612.246873:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495735296/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575612.246881:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575612.246884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575612.246887:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735296:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575612.246893:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495735424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575612.246908:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575612.246916:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff93a258da0e00 x1875453495735424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575612.246933:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495735424/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575612.246938:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575612.246940:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575612.246942:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495735424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575613.206365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575613.206473:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575614.230393:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575614.230462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575615.254409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575615.254411:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575616.278346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575616.278397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575617.302341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575617.302458:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575617.366391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575617.366406:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575617.366412:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575617.366426:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3833 00000100:00080000:1.0:1788575617.366462:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575617.366471:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575617.366516:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495735552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575617.366559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575617.366563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575617.366565:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575617.366570:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3833 00000100:00080000:1.0:1788575617.366573:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575617.366578:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575617.366589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575617.366601:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495735680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575617.366611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575617.366614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575617.366616:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575617.366620:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3833 00000100:00080000:1.0:1788575617.366623:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575617.366628:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575617.366634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575617.366643:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495735808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575617.366652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575617.366660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575617.366662:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575617.366665:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3833 00000100:00080000:1.0:1788575617.366669:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575617.366673:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575617.366677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575617.366686:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495735936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575617.366819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495735552:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575617.366898:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575617.366947:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff93a258da0700 x1875453495735552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.366974:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495735552/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.366985:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.366989:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575617.366993:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495735552:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575617.367000:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575617.367022:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575617.367034:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff93a258da1c00 x1875453495735680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.367046:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495735680/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.367054:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.367057:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575617.367061:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575617.367066:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735808:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575617.367085:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575617.367109:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff93a258da3800 x1875453495735808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.367120:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495735808/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.367139:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.367143:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575617.367146:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495735808:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575617.367152:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495735936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575617.367173:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575617.367184:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff93a258da3480 x1875453495735936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575617.367194:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495735936/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575617.367214:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575617.367218:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575617.367221:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495735936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575618.326346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575618.326402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575619.350388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575619.350465:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575620.374336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575620.374389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575621.398315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575621.398369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575622.422325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575622.422375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575622.486340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.486351:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575622.486355:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.486363:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3838 00000100:00080000:0.0:1788575622.486371:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.486377:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575622.486429:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495736064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575622.486457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.486460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575622.486462:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.486465:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3838 00000100:00080000:0.0:1788575622.486467:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.486470:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575622.486477:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575622.486489:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495736192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575622.486500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.486505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575622.486506:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495736064:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575622.486508:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.486517:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3838 00000100:00080000:0.0:1788575622.486521:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.486524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575622.486530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575622.486541:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495736320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575622.486550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.486553:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575622.486555:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.486564:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3838 00000100:00080000:0.0:1788575622.486569:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.486572:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575622.486576:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788575622.486578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575622.486592:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495736448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575622.486618:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff93a2460d0a80 x1875453495736064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.486637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495736064/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.486647:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.486651:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575622.486654:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495736064:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575622.486676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575622.486698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575622.486710:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff93a2460d0000 x1875453495736192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.486722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495736192/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.486737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.486740:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575622.486742:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575622.486746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736320:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575622.486763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575622.486769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff93a2460d3b80 x1875453495736320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.486776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495736320/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.486781:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.486783:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575622.486784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736320:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575622.486787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495736448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575622.486799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575622.486808:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff93a2460d0700 x1875453495736448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.486819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495736448/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.486826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.486829:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575622.486840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495736448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575623.446305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575623.446368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575624.470415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575624.470627:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575625.494432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575625.494504:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575626.518331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575626.518388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575627.542341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575627.542388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575627.606356:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.606364:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575627.606368:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.606375:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3844 00000100:00080000:0.0:1788575627.606384:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.606391:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575627.606426:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495736576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.606448:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.606450:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575627.606452:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.606454:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3844 00000100:00080000:0.0:1788575627.606456:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.606459:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575627.606478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575627.606491:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495736704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575627.606495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495736576:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575627.606508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.606511:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575627.606513:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.606518:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3844 00000100:00080000:0.0:1788575627.606522:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.606526:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575627.606531:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575627.606542:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495736832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.606550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.606552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575627.606553:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.606558:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3844 00000100:00080000:0.0:1788575627.606562:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.606564:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575627.606567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575627.606570:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575627.606575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495736960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575627.606618:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff93a2460d0700 x1875453495736576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.606637:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495736576/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.606646:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.606650:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575627.606654:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495736576:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575627.606674:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575627.606697:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575627.606709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff93a2460d3b80 x1875453495736704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.606725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495736704/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.606733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.606736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575627.606740:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575627.606746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736832:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575627.606764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575627.606778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff93a2460d0000 x1875453495736832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.606785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495736832/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.606790:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.606792:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575627.606794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495736832:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575627.606797:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495736960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575627.606808:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575627.606813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff93a2460d0a80 x1875453495736960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.606820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495736960/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.606824:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.606825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575627.606827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495736960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575628.566305:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575628.566360:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575629.590358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575629.590413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575630.614384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575630.614440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575631.638344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575631.638400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575632.662406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575632.662464:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788575632.726409:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575632.726421:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575632.726425:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575632.726443:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3849 00000100:00080000:3.0:1788575632.726453:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575632.726461:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575632.726501:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495737088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575632.726530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575632.726534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575632.726536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575632.726540:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3849 00000100:00080000:3.0:1788575632.726543:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575632.726547:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575632.726567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575632.726572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495737088:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575632.726585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495737216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575632.726597:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575632.726601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575632.726602:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575632.726606:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3849 00000100:00080000:3.0:1788575632.726610:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575632.726614:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575632.726645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575632.726653:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575632.726660:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495737344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575632.726671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575632.726675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575632.726676:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575632.726680:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3849 00000100:00080000:3.0:1788575632.726685:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575632.726689:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575632.726694:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575632.726698:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff93a25064d500 x1875453495737088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575632.726708:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495737472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575632.726720:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495737088/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.726737:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.726742:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575632.726746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495737088:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575632.726769:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575632.726793:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575632.726803:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff93a25064c000 x1875453495737216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.726814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495737216/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.726821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.726824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575632.726826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575632.726832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737344:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575632.726850:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575632.726861:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff93a25064f480 x1875453495737344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.726872:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495737344/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.726880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.726883:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575632.726886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737344:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575632.726898:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495737472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575632.726919:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575632.726929:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff93a25064fb80 x1875453495737472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.726941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495737472/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.726948:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.726951:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575632.726954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495737472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575633.686345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575633.686405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575634.710363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575634.710414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575635.734426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575635.734487:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:2.0:1788575636.758317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575636.758362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575637.782375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575637.782699:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575637.846387:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575637.846397:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575637.846402:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575637.846412:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3854 00000100:00080000:3.0:1788575637.846422:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575637.846429:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575637.846463:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495737600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575637.846493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575637.846497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575637.846499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575637.846503:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3854 00000100:00080000:3.0:1788575637.846524:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575637.846529:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575637.846538:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575637.846549:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495737600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575637.846553:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495737728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575637.846566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575637.846576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575637.846578:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575637.846583:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3854 00000100:00080000:3.0:1788575637.846587:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575637.846591:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575637.846596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575637.846606:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495737856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575637.846616:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575637.846618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575637.846623:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575637.846625:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575637.846630:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3854 00000100:00080000:3.0:1788575637.846634:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575637.846638:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575637.846642:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575637.846652:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495737984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575637.846659:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff93a25064e680 x1875453495737600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575637.846680:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495737600/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575637.846689:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575637.846692:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575637.846696:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495737600:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575637.846719:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575637.846742:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575637.846753:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff93a25064ea00 x1875453495737728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575637.846773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495737728/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575637.846780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575637.846782:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575637.846784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575637.846788:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737856:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575637.846803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575637.846810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff93a25064d880 x1875453495737856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575637.846817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495737856/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575637.846821:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575637.846823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575637.846825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495737856:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575637.846828:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495737984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575637.846848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575637.846857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff93a25064dc00 x1875453495737984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575637.846868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495737984/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575637.846880:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575637.846882:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575637.846884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495737984:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575638.806401:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575638.806507:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575639.830374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575639.830436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575640.854311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575640.854371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575641.878350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575641.878416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575642.902371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575642.902446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575642.966404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575642.966412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575642.966415:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575642.966423:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3859 00000100:00080000:2.0:1788575642.966431:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575642.966436:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575642.966465:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495738112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575642.966488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575642.966490:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575642.966491:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575642.966494:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3859 00000100:00080000:2.0:1788575642.966496:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575642.966498:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575642.966505:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575642.966512:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495738240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575642.966519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575642.966522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575642.966524:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575642.966527:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3859 00000100:00080000:2.0:1788575642.966529:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575642.966545:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575642.966549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575642.966558:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495738112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575642.966558:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495738368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575642.966582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575642.966584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575642.966586:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575642.966590:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3859 00000100:00080000:2.0:1788575642.966594:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575642.966598:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575642.966602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575642.966614:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495738496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575642.966663:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575642.966708:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff93a249408a80 x1875453495738112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575642.966722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495738112/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575642.966731:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575642.966734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575642.966737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495738112:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575642.966743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575642.966759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575642.966767:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff93a249409180 x1875453495738240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575642.966774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495738240/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575642.966779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575642.966780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575642.966782:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738240:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575642.966820:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738368:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575642.966836:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575642.966845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff93a24940ad80 x1875453495738368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575642.966866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495738368/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575642.966874:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575642.966877:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575642.966880:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738368:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575642.966887:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495738496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575642.966907:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575642.966919:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff93a24940b800 x1875453495738496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575642.966931:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495738496/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575642.966939:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575642.966942:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575642.966945:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495738496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575643.926369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575643.926420:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575644.950349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575644.950400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575645.974355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575645.974415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575646.998339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575646.998400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575648.022354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575648.022435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575648.086390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.086401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575648.086405:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.086416:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3864 00000100:00080000:3.0:1788575648.086427:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.086434:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575648.086473:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495738624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.086500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.086503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575648.086504:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.086507:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3864 00000100:00080000:3.0:1788575648.086509:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.086511:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575648.086521:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575648.086531:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495738752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.086538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.086540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575648.086541:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.086544:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3864 00000100:00080000:3.0:1788575648.086546:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.086548:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575648.086554:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495738624:192.168.202.102@tcp:250:kworker.0 00000008:00100000:3.0:1788575648.086577:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575648.086592:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495738880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.086605:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.086608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575648.086610:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.086614:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3864 00000100:00080000:3.0:1788575648.086619:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.086623:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575648.086627:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575648.086631:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575648.086640:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495739008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575648.086684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff93a25064ed80 x1875453495738624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.086710:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495738624/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.086720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.086725:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575648.086728:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495738624:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575648.086735:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575648.086755:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575648.086765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff93a25064dc00 x1875453495738752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.086776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495738752/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.086783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.086786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575648.086789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575648.086815:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738880:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575648.086831:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575648.086839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff93a25064d880 x1875453495738880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.086850:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495738880/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.086867:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.086870:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575648.086873:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495738880:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575648.086879:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495739008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575648.086895:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575648.086902:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff93a25064ea00 x1875453495739008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.086912:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495739008/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.086919:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.086921:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575648.086924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495739008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575649.046395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575649.046472:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575650.070345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575650.070438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575651.094414:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575651.094485:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575652.118389:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575652.118453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575653.142367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575653.142423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575653.206413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575653.206431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575653.206435:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575653.206450:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3869 00000100:00080000:3.0:1788575653.206461:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575653.206480:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575653.206531:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495739136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575653.206577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575653.206581:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575653.206583:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575653.206587:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3869 00000100:00080000:3.0:1788575653.206605:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575653.206611:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495739136:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575653.206612:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575653.206634:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575653.206660:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495739264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575653.206666:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575653.206678:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575653.206682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575653.206684:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575653.206690:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3869 00000100:00080000:3.0:1788575653.206694:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575653.206699:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575653.206702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff93a25064e680 x1875453495739136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575653.206706:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575653.206720:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495739136/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575653.206724:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495739392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575653.206730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.206734:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575653.206737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495739136:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575653.206747:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575653.206752:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575653.206754:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575653.206765:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3869 00000100:00100000:1.0:1788575653.206766:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739264:192.168.202.102@tcp:38:kworker.0 00000100:00080000:3.0:1788575653.206773:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575653.206779:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575653.206785:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575653.206789:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575653.206797:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff93a25064fb80 x1875453495739264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575653.206806:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495739520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575653.206808:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495739264/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.206823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.206825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575653.206828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575653.206838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739392:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575653.206854:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575653.206863:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff93a25064f480 x1875453495739392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575653.206871:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495739392/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.206875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.206877:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575653.206879:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739392:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575653.206886:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495739520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575653.206897:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575653.206904:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff93a25064c000 x1875453495739520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575653.206917:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495739520/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.206922:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.206924:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575653.206926:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495739520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575654.166375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575654.166447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575655.190377:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575655.190402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575656.214405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575656.214415:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575657.238423:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575657.238521:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575658.262552:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575658.262858:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788575658.326358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.326368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575658.326373:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.326382:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3874 00000100:00080000:0.0:1788575658.326415:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.326423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575658.326460:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495739648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575658.326493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.326496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575658.326498:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.326501:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3874 00000100:00080000:0.0:1788575658.326504:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.326507:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575658.326515:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575658.326524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495739776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575658.326530:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.326532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575658.326533:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.326536:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3874 00000100:00080000:0.0:1788575658.326537:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.326539:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575658.326541:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575658.326546:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495739904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575658.326551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.326557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575658.326558:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.326561:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3874 00000100:00080000:0.0:1788575658.326563:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.326565:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575658.326568:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575658.326573:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495740032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575658.326649:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495739648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575658.326724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575658.326740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739904:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575658.326751:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495740032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575659.286437:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575659.287645:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575660.310485:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575660.311133:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575661.336828:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575661.336858:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575661.398557:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788575661.398629:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575661.398646:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575661.398697:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575661.398706:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575661.398714:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575661.399394:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff93a2460d0380 x1875453495739648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.399420:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495739648/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.399433:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.399439:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575661.399445:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495739648:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575661.399455:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff93a2460d1500 x1875453495739776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.399469:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495739776/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.399486:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.399489:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575661.399492:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575661.399499:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff93a2460d2680 x1875453495739904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.399509:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495739904/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.399516:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.399518:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575661.399521:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495739904:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575661.399527:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff93a2460d3480 x1875453495740032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.399536:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495740032/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.399542:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.399545:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575661.399547:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495740032:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788575662.358963:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575662.360458:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575663.382344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575663.382436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575663.446390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.446401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575663.446405:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.446416:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3879 00000100:00080000:2.0:1788575663.446425:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.446431:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575663.446465:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495740160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575663.446490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.446494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575663.446495:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.446498:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3879 00000100:00080000:2.0:1788575663.446502:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.446505:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575663.446514:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575663.446524:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495740288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575663.446533:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.446548:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575663.446551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.446556:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3879 00000100:00080000:2.0:1788575663.446559:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.446563:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575663.446567:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575663.446586:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495740416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575663.446591:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495740160:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575663.446601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.446607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575663.446608:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.446612:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3879 00000100:00080000:2.0:1788575663.446615:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.446619:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575663.446622:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575663.446635:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495740544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575663.446701:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575663.446764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff93a249408e00 x1875453495740160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575663.446810:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495740160/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575663.446823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575663.446828:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575663.446832:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495740160:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575663.446841:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575663.446868:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575663.446881:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff93a249409c00 x1875453495740288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575663.446897:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495740288/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575663.446906:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575663.446909:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575663.446913:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575663.446948:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740416:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575663.446977:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575663.446990:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff93a24940a680 x1875453495740416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575663.447000:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495740416/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575663.447033:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575663.447037:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575663.447040:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740416:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575663.447046:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495740544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575663.447070:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575663.447080:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff93a24940b800 x1875453495740544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575663.447089:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495740544/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575663.447096:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575663.447099:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575663.447101:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495740544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575664.406433:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575664.406486:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575665.430312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575665.430475:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575666.454362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575666.454418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575667.478315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575667.478378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575668.502338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575668.502439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575668.566421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.566432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575668.566435:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.566445:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3885 00000100:00080000:0.0:1788575668.566455:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.566461:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575668.566496:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495740672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.566525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.566528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575668.566530:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.566533:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3885 00000100:00080000:0.0:1788575668.566535:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.566538:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575668.566545:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575668.566570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495740800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.566579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.566582:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575668.566584:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.566587:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3885 00000100:00080000:0.0:1788575668.566591:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.566595:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575668.566600:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575668.566607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495740928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.566613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.566615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575668.566617:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.566620:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3885 00000100:00080000:0.0:1788575668.566635:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.566639:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575668.566643:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575668.566654:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495741056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575668.566741:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495740672:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788575668.566795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575668.566831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff93a2460d2300 x1875453495740672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575668.566848:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495740672/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575668.566856:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575668.566860:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575668.566864:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495740672:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575668.566878:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575668.566901:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575668.566914:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff93a2460d1f80 x1875453495740800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575668.566926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495740800/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575668.566933:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575668.566936:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575668.566939:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575668.566945:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740928:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575668.566965:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575668.566976:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff93a2460d2a00 x1875453495740928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575668.566996:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495740928/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575668.567003:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575668.567006:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575668.567010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495740928:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575668.567016:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495741056:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575668.567036:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575668.567047:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff93a2460d1180 x1875453495741056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575668.567066:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495741056/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575668.567074:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575668.567077:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575668.567080:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495741056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575669.526324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575669.526374:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575670.550359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575670.550416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575671.574344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575671.574389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575672.598352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575672.598409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575673.622350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575673.622423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575673.686378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575673.686388:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575673.686393:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575673.686401:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3890 00000100:00080000:1.0:1788575673.686410:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575673.686415:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575673.686448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495741184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575673.686475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575673.686479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575673.686481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575673.686485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3890 00000100:00080000:1.0:1788575673.686489:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575673.686493:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575673.686503:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575673.686509:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495741184:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575673.686518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495741312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575673.686548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575673.686552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575673.686554:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575673.686559:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3890 00000100:00080000:1.0:1788575673.686564:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575673.686568:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575673.686573:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575673.686574:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575673.686586:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495741440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575673.686594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575673.686596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575673.686597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575673.686600:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3890 00000100:00080000:1.0:1788575673.686602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575673.686605:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575673.686610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575673.686617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff93a258da3100 x1875453495741184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575673.686623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495741568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575673.686652:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495741184/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575673.686664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575673.686669:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575673.686674:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495741184:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575673.686696:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741312:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575673.686719:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575673.686732:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff93a258da2a00 x1875453495741312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575673.686744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495741312/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575673.686752:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575673.686755:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575673.686759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575673.686765:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741440:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575673.686785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575673.686795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff93a258da1880 x1875453495741440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575673.686806:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495741440/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575673.686826:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575673.686830:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575673.686833:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741440:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575673.686839:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495741568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575673.686860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575673.686870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff93a258da1180 x1875453495741568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575673.686880:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495741568/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575673.686887:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575673.686890:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575673.686893:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495741568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575674.646346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575674.646399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575675.670335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575675.670388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575676.694444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575676.694882:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575677.718344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575677.718408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575678.742357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575678.742409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575678.806347:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575678.806358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575678.806362:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575678.806371:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3895 00000100:00080000:1.0:1788575678.806397:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575678.806405:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575678.806438:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495741696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575678.806469:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575678.806472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575678.806474:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575678.806478:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3895 00000100:00080000:1.0:1788575678.806482:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575678.806486:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575678.806496:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575678.806510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495741824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575678.806512:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495741696:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575678.806528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575678.806544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575678.806546:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575678.806552:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3895 00000100:00080000:1.0:1788575678.806568:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575678.806572:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788575678.806578:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788575678.806579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575678.806595:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495741952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575678.806605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575678.806615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575678.806617:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575678.806621:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3895 00000100:00080000:1.0:1788575678.806626:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575678.806633:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575678.806634:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff93a258da0380 x1875453495741696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575678.806639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575678.806657:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495742080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575678.806659:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495741696/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.806672:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.806675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575678.806681:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495741696:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575678.806711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575678.806736:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575678.806748:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff93a258da0000 x1875453495741824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.806757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495741824/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.806763:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.806765:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575678.806767:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575678.806770:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741952:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575678.806784:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575678.806790:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff93a258da2300 x1875453495741952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.806796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495741952/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.806801:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.806802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575678.806804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495741952:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575678.806807:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495742080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575678.806816:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575678.806821:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff93a258da1180 x1875453495742080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.806833:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495742080/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.806838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.806839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575678.806841:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495742080:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575679.766318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575679.766392:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575680.790360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575680.790435:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575681.814365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575681.814411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575682.838359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575682.840331:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575683.862355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575683.862430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575683.926363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575683.926374:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575683.926378:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575683.926388:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3900 00000100:00080000:1.0:1788575683.926398:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575683.926406:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575683.926457:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495742208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575683.926489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575683.926492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575683.926494:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575683.926498:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3900 00000100:00080000:1.0:1788575683.926502:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575683.926506:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575683.926515:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575683.926529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495742336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575683.926538:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495742208:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575683.926556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575683.926559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575683.926561:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575683.926566:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3900 00000100:00080000:1.0:1788575683.926586:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575683.926590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575683.926595:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575683.926602:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575683.926617:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495742464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575683.926627:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575683.926630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575683.926632:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575683.926637:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff93a258da1880 x1875453495742208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575683.926639:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3900 00000100:00080000:1.0:1788575683.926653:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575683.926659:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575683.926660:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495742208/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575683.926666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1788575683.926677:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788575683.926682:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495742592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575683.926684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575683.926693:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495742208:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575683.926724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575683.926747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575683.926761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff93a258da2a00 x1875453495742336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.926772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495742336/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.926802:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.926806:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575683.926809:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575683.926815:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742464:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575683.926838:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575683.926849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff93a258da3100 x1875453495742464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.926859:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495742464/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.926866:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.926869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575683.926871:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742464:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575683.926877:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495742592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575683.926895:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575683.926904:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff93a258da3480 x1875453495742592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.926926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495742592/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.926934:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.926937:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575683.926946:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495742592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575684.886375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575684.886433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575685.910360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575685.910457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575686.934329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575686.934381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575687.958340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575687.958397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575688.982323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575688.982362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575689.046411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575689.046421:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575689.046425:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575689.046432:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3905 00000100:00080000:3.0:1788575689.046441:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575689.046445:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575689.046491:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495742720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575689.046517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575689.046521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575689.046523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575689.046526:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3905 00000100:00080000:3.0:1788575689.046529:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575689.046533:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575689.046542:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575689.046555:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495742848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575689.046564:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575689.046568:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575689.046571:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575689.046572:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495742720:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575689.046577:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3905 00000100:00080000:3.0:1788575689.046584:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575689.046602:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575689.046609:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575689.046621:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495742976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575689.046630:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575689.046633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575689.046635:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575689.046640:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575689.046641:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3905 00000100:00080000:3.0:1788575689.046648:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575689.046652:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575689.046656:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575689.046667:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495743104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575689.046686:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff93a25064d500 x1875453495742720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575689.046706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495742720/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575689.046716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575689.046719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575689.046723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495742720:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575689.046745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575689.046766:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575689.046778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff93a25064f100 x1875453495742848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575689.046790:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495742848/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575689.046797:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575689.046801:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575689.046804:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742848:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575689.046810:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742976:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575689.046840:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575689.046858:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff93a25064ce00 x1875453495742976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575689.046869:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495742976/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575689.046877:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575689.046880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575689.046883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495742976:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575689.046888:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495743104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575689.046917:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575689.046928:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff93a25064d180 x1875453495743104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575689.046938:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495743104/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575689.046945:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575689.046948:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575689.046951:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495743104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575690.006343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575690.006398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575691.030334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575691.030389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575692.054347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575692.054422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575693.078343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575693.078399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575694.102336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575694.102405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575694.166374:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575694.166383:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575694.166388:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575694.166398:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3910 00000100:00080000:2.0:1788575694.166408:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575694.166415:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575694.166448:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495743232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575694.166492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575694.166496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575694.166497:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575694.166502:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3910 00000100:00080000:2.0:1788575694.166505:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575694.166509:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575694.166517:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575694.166534:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495743232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575694.166534:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495743360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575694.166548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575694.166552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575694.166554:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575694.166558:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3910 00000100:00080000:2.0:1788575694.166561:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575694.166565:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575694.166570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575694.166580:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495743488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575694.166589:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575694.166590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575694.166596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575694.166597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575694.166602:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3910 00000100:00080000:2.0:1788575694.166613:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575694.166617:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575694.166623:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575694.166624:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff93a24940ad80 x1875453495743232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575694.166637:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495743616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575694.166641:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495743232/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575694.166657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575694.166660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575694.166663:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495743232:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575694.166681:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495743360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575694.166697:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575694.166705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff93a249409180 x1875453495743360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575694.166712:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495743360/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575694.166717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575694.166719:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575694.166720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495743360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575694.166724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495743488:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575694.166733:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575694.166738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff93a249408a80 x1875453495743488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575694.166744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495743488/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575694.166749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575694.166751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575694.166752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495743488:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575694.166760:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495743616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575694.166772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575694.166778:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff93a249408700 x1875453495743616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575694.166785:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495743616/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575694.166789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575694.166791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575694.166792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495743616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575695.126384:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575695.126467:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575696.150361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575696.150456:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575697.174340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575697.174396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575698.198348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575698.198411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575699.222359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575699.222431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575699.286367:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575699.286382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575699.286387:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575699.286420:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3915 00000100:00080000:3.0:1788575699.286431:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575699.286438:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575699.286473:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495743744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575699.286497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575699.286500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575699.286501:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575699.286504:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3915 00000100:00080000:3.0:1788575699.286506:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575699.286508:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575699.286517:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575699.286525:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495743872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575699.286532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575699.286534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575699.286535:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575699.286540:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3915 00000100:00080000:3.0:1788575699.286543:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575699.286546:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575699.286552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575699.286553:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495743744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575699.286566:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495744000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575699.286607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575699.286611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575699.286614:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575699.286618:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3915 00000100:00080000:3.0:1788575699.286623:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575699.286628:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575699.286639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575699.286647:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575699.286653:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495744128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575699.286702:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff93a25064c700 x1875453495743744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575699.286722:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495743744/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575699.286733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575699.286738:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575699.286741:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495743744:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575699.286758:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495743872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575699.286783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575699.286795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff93a25064ca80 x1875453495743872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575699.286814:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495743872/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575699.286822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575699.286825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575699.286828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495743872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575699.286857:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744000:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575699.286880:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575699.286892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff93a25064c380 x1875453495744000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575699.286902:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495744000/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575699.286910:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575699.286914:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575699.286917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744000:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575699.286922:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495744128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575699.286941:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575699.286951:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff93a25064e300 x1875453495744128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575699.286962:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495744128/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575699.286981:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575699.286985:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575699.286988:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495744128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575700.246388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575700.246501:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575701.270336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575701.271437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575702.294356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575702.294410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575703.318370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575703.318428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575704.342330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575704.342396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575704.406346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575704.406355:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575704.406359:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575704.406367:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3920 00000100:00080000:3.0:1788575704.406374:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575704.406379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575704.406408:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495744256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575704.406430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575704.406433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575704.406445:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575704.406450:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3920 00000100:00080000:3.0:1788575704.406453:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575704.406458:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575704.406467:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575704.406484:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495744256:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575704.406485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495744384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575704.406502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575704.406504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575704.406505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575704.406508:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3920 00000100:00080000:3.0:1788575704.406511:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575704.406514:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575704.406518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575704.406527:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495744512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575704.406533:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575704.406536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575704.406537:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575704.406539:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3920 00000100:00080000:3.0:1788575704.406541:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575704.406543:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575704.406546:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575704.406548:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575704.406559:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495744640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575704.406608:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff93a25064df80 x1875453495744256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575704.406627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495744256/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575704.406638:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575704.406642:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575704.406646:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495744256:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575704.406675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575704.406696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575704.406720:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff93a25064e300 x1875453495744384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575704.406733:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495744384/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575704.406741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575704.406744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575704.406748:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575704.406760:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744512:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575704.406779:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575704.406789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff93a25064c380 x1875453495744512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575704.406801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495744512/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575704.406809:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575704.406812:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575704.406816:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744512:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575704.406821:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495744640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575704.406839:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575704.406849:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff93a25064ca80 x1875453495744640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575704.406868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495744640/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575704.406877:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575704.406880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575704.406883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495744640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575705.366351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575705.366409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575706.390326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575706.390417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575707.414379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575707.415336:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575708.439998:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575708.440412:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575709.462448:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575709.462481:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:1.0:1788575709.526471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575709.526482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575709.526486:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575709.526495:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3925 00000100:00080000:1.0:1788575709.526535:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575709.526543:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575709.526597:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495744768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575709.526640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575709.526644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575709.526646:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575709.526651:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3925 00000100:00080000:1.0:1788575709.526655:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575709.526666:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575709.526675:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575709.526688:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495744896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575709.526697:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575709.526700:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575709.526701:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575709.526705:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3925 00000100:00080000:1.0:1788575709.526717:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575709.526721:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575709.526726:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575709.526736:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495745024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575709.526752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575709.526761:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575709.526763:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575709.526767:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3925 00000100:00080000:1.0:1788575709.526771:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575709.526775:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575709.526790:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575709.526802:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495745152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575709.526832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495744768:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575709.526889:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575709.526922:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff93a258da3800 x1875453495744768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575709.526943:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495744768/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575709.526953:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575709.526956:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575709.526959:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495744768:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575709.526967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575709.526985:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575709.526995:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff93a258da1c00 x1875453495744896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575709.527006:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495744896/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575709.527013:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575709.527016:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575709.527019:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495744896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575709.527024:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745024:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575709.527040:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575709.527049:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff93a258da0700 x1875453495745024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575709.527058:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495745024/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575709.527075:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575709.527078:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575709.527081:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745024:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575709.527087:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495745152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575709.527104:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575709.527113:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff93a258da0e00 x1875453495745152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575709.527122:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495745152/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575709.527129:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575709.527132:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575709.527135:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495745152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575710.486335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575710.486432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575711.510376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575711.510426:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575712.534345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575712.534402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575713.558430:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575713.558483:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575714.582380:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575714.582444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575714.646484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575714.646496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575714.646500:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575714.646510:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3931 00000100:00080000:3.0:1788575714.646520:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575714.646533:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575714.646566:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495745280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575714.646779:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495745280:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575714.646871:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575714.646938:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575714.646943:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575714.646945:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575714.646950:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3931 00000100:00080000:3.0:1788575714.646954:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575714.646958:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575714.646981:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575714.646994:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495745408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575714.647118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575714.647122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575714.647124:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575714.647128:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3931 00000100:00080000:3.0:1788575714.647133:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575714.647137:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575714.647143:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575714.647161:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff93a25064c700 x1875453495745280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575714.647161:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495745536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575714.647193:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575714.647197:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575714.647200:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575714.647201:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495745280/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575714.647208:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3931 00000100:00080000:3.0:1788575714.647215:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575714.647220:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575714.647221:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575714.647256:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575714.647268:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495745664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575714.647270:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575714.647278:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495745280:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575714.647324:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575714.647349:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575714.647363:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff93a25064d180 x1875453495745408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575714.647462:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495745408/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575714.647472:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575714.647475:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575714.647479:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575714.647485:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745536:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575714.647505:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575714.647515:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff93a25064ce00 x1875453495745536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575714.647532:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495745536/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575714.647540:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575714.647543:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575714.647546:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745536:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575714.647552:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495745664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575714.647571:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575714.647581:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff93a25064f100 x1875453495745664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575714.647598:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495745664/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575714.647605:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575714.647608:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575714.647611:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495745664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575715.606380:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575715.606428:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575716.630376:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575716.631029:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575717.654324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575717.654367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575718.678331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575718.678382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575719.702316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575719.702365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575719.766355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575719.766363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575719.766367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575719.766375:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3936 00000100:00080000:0.0:1788575719.766383:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575719.766389:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575719.766420:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495745792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575719.766445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575719.766447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575719.766449:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575719.766452:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3936 00000100:00080000:0.0:1788575719.766454:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575719.766456:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575719.766464:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575719.766471:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495745920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575719.766495:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495745792:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575719.766496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575719.766502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575719.766505:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575719.766510:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3936 00000100:00080000:0.0:1788575719.766512:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575719.766515:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575719.766518:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575719.766526:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495746048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575719.766533:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575719.766535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575719.766537:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575719.766539:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3936 00000100:00080000:0.0:1788575719.766549:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575719.766551:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575719.766553:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575719.766559:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575719.766561:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495746176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575719.766595:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff93a2460d1180 x1875453495745792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575719.766607:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495745792/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575719.766614:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575719.766617:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575719.766620:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495745792:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575719.766637:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575719.766650:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575719.766657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff93a2460d2a00 x1875453495745920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575719.766664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495745920/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575719.766674:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575719.766676:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575719.766678:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495745920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575719.766687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746048:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575719.766701:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575719.766707:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff93a2460d1f80 x1875453495746048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575719.766714:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495746048/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575719.766718:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575719.766720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575719.766722:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746048:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575719.766725:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495746176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575719.766733:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575719.766738:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff93a2460d2300 x1875453495746176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575719.766744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495746176/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575719.766749:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575719.766751:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575719.766759:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495746176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575720.726325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575720.726400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575721.750332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575721.750385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575722.774333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575722.774380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575723.798460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575723.798778:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575724.822664:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575724.824011:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575724.886385:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575724.886400:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575724.886406:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575724.886418:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3941 00000100:00080000:2.0:1788575724.886432:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575724.886443:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575724.886492:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495746304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575724.886520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575724.886524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575724.886526:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575724.886530:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3941 00000100:00080000:2.0:1788575724.886534:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575724.886539:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575724.886550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575724.886563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495746432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575724.886575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575724.886580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575724.886581:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495746304:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575724.886582:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575724.886607:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3941 00000100:00080000:2.0:1788575724.886612:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575724.886616:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575724.886622:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575724.886634:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495746560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575724.886650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575724.886653:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575724.886655:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575724.886659:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3941 00000100:00080000:2.0:1788575724.886663:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575724.886666:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575724.886671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575724.886681:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495746688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575724.886713:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575724.886745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746560:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575724.886772:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495746688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575725.846688:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575725.847795:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575726.870355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575726.870443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575727.895774:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575727.896114:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788575727.958576:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788575727.958639:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575727.958660:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575727.958712:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575727.958734:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575727.958744:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575727.958872:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575727] req@ffff93a249408000 x1875453495746432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575727.958899:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495746432/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575727.958913:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575727.958919:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575727.958924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575727.958932:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575727] req@ffff93a24940aa00 x1875453495746560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575727.958943:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495746560/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575727.958950:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575727.958953:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575727.958955:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746560:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575727.958963:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575727] req@ffff93a249409500 x1875453495746688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575727.958996:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495746688/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575727.959003:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575727.959007:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575727.959010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495746688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575727.959017:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575727] req@ffff93a249409880 x1875453495746304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575727.959028:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495746304/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575727.959036:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575727.959039:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575727.959043:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495746304:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575728.918888:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575728.918920:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575729.942381:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575729.942452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575730.006420:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575730.006434:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575730.006440:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575730.006456:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3946 00000100:00080000:3.0:1788575730.006467:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575730.006474:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575730.006514:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495746816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575730.006569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575730.006573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575730.006594:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575730.006598:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3946 00000100:00080000:3.0:1788575730.006602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575730.006604:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575730.006636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575730.006649:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495746944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575730.006660:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575730.006663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575730.006665:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575730.006670:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3946 00000100:00080000:3.0:1788575730.006673:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575730.006678:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575730.006683:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575730.006694:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495747072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575730.006702:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575730.006704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575730.006706:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575730.006708:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3946 00000100:00080000:3.0:1788575730.006711:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575730.006714:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575730.006718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575730.006736:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495747200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575730.007445:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495746816:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788575730.007554:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575730.007636:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff93a25064d500 x1875453495746816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575730.007661:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495746816/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575730.007676:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575730.007680:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575730.007685:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495746816:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575730.007703:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575730.007730:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575730.007743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff93a25064c000 x1875453495746944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575730.007757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495746944/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575730.007765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575730.007769:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575730.007773:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495746944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575730.007781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747072:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575730.007804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575730.007815:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff93a25064f480 x1875453495747072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575730.007827:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495747072/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575730.007835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575730.007838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575730.007841:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747072:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575730.007848:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495747200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575730.007869:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575730.007880:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff93a25064fb80 x1875453495747200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575730.007898:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495747200/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575730.007904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575730.007906:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575730.007908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495747200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575730.966375:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575730.966470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575731.990340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575731.990575:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575733.014367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575733.014703:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575734.038353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575734.038824:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575735.062351:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575735.062408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575735.126498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575735.126509:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575735.126513:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575735.126523:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3951 00000100:00080000:3.0:1788575735.126532:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575735.126538:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575735.126574:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495747328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575735.126604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575735.126609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575735.126611:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575735.126616:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3951 00000100:00080000:3.0:1788575735.126619:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575735.126624:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575735.126634:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575735.126647:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495747456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575735.126656:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575735.126671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575735.126674:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575735.126679:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3951 00000100:00080000:3.0:1788575735.126682:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575735.126686:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575735.126692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575735.126703:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495747584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575735.126714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575735.126717:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575735.126720:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575735.126724:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3951 00000100:00080000:3.0:1788575735.126727:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575735.126731:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575735.126735:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575735.126738:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495747328:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575735.126748:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495747712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575735.126803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575735.126839:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff93a25064e680 x1875453495747328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575735.126854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495747328/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575735.126864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575735.126868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575735.126872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495747328:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575735.126877:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575735.126895:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575735.126905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff93a25064ea00 x1875453495747456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575735.126915:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495747456/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575735.126933:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575735.126936:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575735.126953:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575735.126958:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747584:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575735.126980:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575735.126989:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff93a25064d880 x1875453495747584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575735.126998:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495747584/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575735.127005:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575735.127007:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575735.127010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747584:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575735.127035:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495747712:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575735.127061:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575735.127071:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff93a25064dc00 x1875453495747712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575735.127091:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495747712/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575735.127099:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575735.127102:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575735.127110:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495747712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575736.086349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575736.086417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575737.110348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575737.111320:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575738.134365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575738.134460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575739.158366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575739.158453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575740.182401:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575740.182496:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575740.246393:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575740.246407:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575740.246412:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575740.246426:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3956 00000100:00080000:2.0:1788575740.246440:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575740.246449:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575740.246497:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495747840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575740.246531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575740.246535:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575740.246536:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575740.246541:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3956 00000100:00080000:2.0:1788575740.246544:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575740.246549:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575740.246561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575740.246575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495747968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575740.246582:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495747840:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575740.246591:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575740.246596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575740.246598:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575740.246603:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3956 00000100:00080000:2.0:1788575740.246625:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575740.246630:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575740.246637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575740.246651:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495748096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575740.246662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575740.246666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575740.246668:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575740.246670:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575740.246676:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3956 00000100:00080000:2.0:1788575740.246681:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575740.246686:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575740.246692:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575740.246703:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495748224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575740.246726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff93a24940b480 x1875453495747840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575740.246749:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495747840/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575740.246761:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575740.246766:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575740.246771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495747840:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575740.246798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747968:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575740.246830:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575740.246843:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff93a24940bb80 x1875453495747968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575740.246855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495747968/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575740.246864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575740.246867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575740.246870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495747968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575740.246876:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748096:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575740.246892:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575740.246907:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff93a24940b100 x1875453495748096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575740.246914:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495748096/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575740.246919:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575740.246921:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575740.246923:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748096:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575740.246926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495748224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575740.246938:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575740.246943:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff93a249409880 x1875453495748224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575740.247019:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495748224/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575740.247029:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575740.247032:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575740.247035:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495748224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575741.206317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575741.206383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575742.230332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575742.230380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575743.254291:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575743.254361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575744.278367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575744.278437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575745.302361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575745.302424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575745.366393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575745.366408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575745.366414:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575745.366427:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3961 00000100:00080000:0.0:1788575745.366464:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575745.366472:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575745.366519:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495748352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575745.366557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575745.366561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575745.366564:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575745.366568:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3961 00000100:00080000:0.0:1788575745.366572:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575745.366577:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575745.366599:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575745.366614:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495748352:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575745.366616:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495748480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575745.366632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575745.366635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575745.366637:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575745.366643:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3961 00000100:00080000:0.0:1788575745.366647:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575745.366651:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575745.366657:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575745.366670:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495748608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575745.366681:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575745.366692:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1788575745.366693:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575745.366696:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575745.366703:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3961 00000100:00080000:0.0:1788575745.366708:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575745.366713:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575745.366718:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575745.366732:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495748736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575745.366739:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff93a2460d3480 x1875453495748352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575745.366766:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495748352/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575745.366785:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575745.366790:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575745.366794:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495748352:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575745.366822:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575745.366841:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575745.366850:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff93a2460d2680 x1875453495748480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575745.366858:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495748480/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575745.366876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575745.366880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575745.366883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575745.366889:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748608:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575745.366905:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575745.366912:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff93a2460d1500 x1875453495748608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575745.366919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495748608/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575745.366924:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575745.366926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575745.366928:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748608:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575745.366937:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495748736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575745.366949:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575745.366955:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff93a2460d0380 x1875453495748736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575745.366962:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495748736/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575745.366967:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575745.366969:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575745.366971:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495748736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575746.326323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575746.326539:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575747.350355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575747.350436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575748.374333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575748.374378:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575749.398304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575749.398383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575750.422322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575750.422377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575750.486317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575750.486326:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575750.486330:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575750.486339:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3966 00000100:00080000:0.0:1788575750.486349:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575750.486356:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575750.486389:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495748864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575750.486417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575750.486420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575750.486422:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575750.486426:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3966 00000100:00080000:0.0:1788575750.486429:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575750.486432:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575750.486441:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575750.486452:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495748992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575750.486462:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495748864:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575750.486479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575750.486481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575750.486483:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575750.486486:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3966 00000100:00080000:0.0:1788575750.486489:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575750.486492:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575750.486496:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575750.486506:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495749120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575750.486512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575750.486514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575750.486515:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575750.486517:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3966 00000100:00080000:0.0:1788575750.486521:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575750.486523:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575750.486525:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575750.486533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495749248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575750.486533:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575750.486575:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff93a2460d0a80 x1875453495748864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.486593:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495748864/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.486603:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.486605:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575750.486608:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495748864:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575750.486641:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575750.486665:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575750.486676:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff93a2460d0000 x1875453495748992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.486697:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495748992/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.486705:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.486707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575750.486709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495748992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575750.486713:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495749120:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575750.486730:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575750.486737:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff93a2460d3b80 x1875453495749120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.486744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495749120/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.486748:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.486750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575750.486752:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495749120:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575750.486755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495749248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575750.486764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575750.486769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff93a2460d0700 x1875453495749248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.486775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495749248/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.486788:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.486791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575750.486793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495749248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575751.446340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575751.446443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575752.470329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575752.470377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575753.494318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575753.494373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575754.518332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575754.520313:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575755.542327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575755.542396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575755.606383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.606393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575755.606397:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.606407:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3972 00000100:00080000:2.0:1788575755.606417:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.606423:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575755.606471:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495749376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575755.606496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.606500:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575755.606501:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.606506:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3972 00000100:00080000:2.0:1788575755.606509:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.606513:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575755.606526:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575755.606539:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495749504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575755.606551:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495749376:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575755.606552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.606559:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575755.606562:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.606567:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3972 00000100:00080000:2.0:1788575755.606570:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.606590:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575755.606596:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575755.606608:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495749632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575755.606617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575755.606619:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575755.606623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575755.606626:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.606631:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3972 00000100:00080000:2.0:1788575755.606636:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.606640:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575755.606644:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575755.606655:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495749760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575755.606666:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff93a249409500 x1875453495749376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575755.606686:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495749376/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575755.606696:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575755.606700:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575755.606703:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495749376:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575755.606725:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495749504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575755.606747:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575755.606759:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff93a24940aa00 x1875453495749504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575755.606770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495749504/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575755.606778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575755.606781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575755.606784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495749504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575755.606800:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495749632:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575755.606820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575755.606831:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff93a249408000 x1875453495749632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575755.606841:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495749632/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575755.606848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575755.606851:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575755.606855:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495749632:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575755.606860:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495749760:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575755.606879:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575755.606889:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff93a249408700 x1875453495749760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575755.606900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495749760/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575755.606907:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575755.606910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575755.606912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495749760:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575756.566370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575756.566436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575757.590322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575757.590361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575758.614345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575758.614449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575759.638327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575759.638419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575760.662301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575760.662361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575760.726360:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575760.726369:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575760.726372:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575760.726382:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3977 00000100:00080000:2.0:1788575760.726391:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575760.726404:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575760.726437:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495749888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575760.726466:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575760.726469:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575760.726470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575760.726473:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3977 00000100:00080000:2.0:1788575760.726476:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575760.726478:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575760.726487:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575760.726496:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495750016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575760.726502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495749888:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575760.726504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575760.726510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575760.726512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575760.726516:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3977 00000100:00080000:2.0:1788575760.726518:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575760.726520:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575760.726522:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575760.726530:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495750144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575760.726537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575760.726539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575760.726540:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575760.726542:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3977 00000100:00080000:2.0:1788575760.726549:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575760.726551:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575760.726554:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575760.726561:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495750272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575760.726568:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575760.726620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff93a249408a80 x1875453495749888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.726645:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495749888/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.726663:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.726667:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575760.726670:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495749888:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575760.726694:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750016:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575760.726715:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575760.726727:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff93a249409180 x1875453495750016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.726738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495750016/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.726745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.726748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575760.726750:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750016:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575760.726755:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750144:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575760.726774:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575760.726785:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff93a24940ad80 x1875453495750144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.726796:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495750144/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.726813:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.726817:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575760.726820:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750144:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575760.726825:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495750272:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575760.726846:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575760.726857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff93a24940b800 x1875453495750272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.726868:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495750272/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.726875:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.726878:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575760.726881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495750272:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575761.686339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575761.686398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575762.710331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575762.710390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575763.734348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575763.734413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575764.758335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575764.758403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575765.782359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575765.782422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575765.846329:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575765.846339:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575765.846344:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575765.846353:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3982 00000100:00080000:3.0:1788575765.846363:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575765.846370:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575765.846414:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ed80 x1875453495750400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575765.846457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575765.846460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575765.846461:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575765.846464:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3982 00000100:00080000:3.0:1788575765.846467:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575765.846469:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575765.846478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575765.846488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064dc00 x1875453495750528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575765.846494:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575765.846496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575765.846498:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575765.846501:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3982 00000100:00080000:3.0:1788575765.846502:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575765.846504:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575765.846507:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575765.846513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d880 x1875453495750656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575765.846520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575765.846523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575765.846524:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575765.846526:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3982 00000100:00080000:3.0:1788575765.846532:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575765.846536:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495750400:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788575765.846537:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575765.846543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575765.846551:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ea00 x1875453495750784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575765.846612:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575765.846661:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff93a25064ed80 x1875453495750400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575765.846677:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ed80 x1875453495750400/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575765.846686:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575765.846689:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575765.846692:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495750400:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575765.846704:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750528:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575765.846721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575765.846729:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff93a25064dc00 x1875453495750528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575765.846736:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064dc00 x1875453495750528/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575765.846741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575765.846743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575765.846745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750528:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575765.846775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750656:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575765.846787:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575765.846793:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff93a25064d880 x1875453495750656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575765.846799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d880 x1875453495750656/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575765.846804:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575765.846806:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575765.846808:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495750656:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575765.846811:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495750784:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575765.846820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575765.846825:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff93a25064ea00 x1875453495750784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575765.846837:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ea00 x1875453495750784/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575765.846842:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575765.846844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575765.846845:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495750784:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575766.806341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575766.806468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575767.830300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575767.830364:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575768.854326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575768.854370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575769.878333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575769.878388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575770.902310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575770.902356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575770.966329:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575770.966341:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575770.966345:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575770.966353:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3987 00000100:00080000:0.0:1788575770.966382:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575770.966387:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575770.966420:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495750912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575770.966440:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575770.966443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575770.966444:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575770.966447:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3987 00000100:00080000:0.0:1788575770.966449:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575770.966451:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575770.966457:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575770.966465:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495751040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575770.966472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575770.966474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575770.966475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575770.966478:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3987 00000100:00080000:0.0:1788575770.966480:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575770.966484:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495750912:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575770.966485:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575770.966490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575770.966497:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495751168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575770.966504:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575770.966510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575770.966511:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575770.966514:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3987 00000100:00080000:0.0:1788575770.966529:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575770.966532:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575770.966535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575770.966537:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575770.966545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495751296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575770.966579:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff93a2460d0700 x1875453495750912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575770.966598:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495750912/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575770.966608:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575770.966611:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575770.966625:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495750912:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575770.966631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751040:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575770.966648:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575770.966657:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff93a2460d3b80 x1875453495751040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575770.966664:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495751040/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575770.966678:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575770.966680:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575770.966682:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751040:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575770.966701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751168:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575770.966713:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575770.966719:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff93a2460d0000 x1875453495751168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575770.966726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495751168/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575770.966730:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575770.966732:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575770.966733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751168:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575770.966737:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495751296:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575770.966753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575770.966759:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff93a2460d0a80 x1875453495751296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575770.966765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495751296/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575770.966770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575770.966771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575770.966778:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495751296:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575771.926326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575771.926368:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575772.950318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575772.950362:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575773.974357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575773.974413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575774.998314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575774.998387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575776.022417:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575776.022544:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575776.086415:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575776.086429:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575776.086435:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575776.086447:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3992 00000100:00080000:2.0:1788575776.086460:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575776.086470:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575776.086518:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495751424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575776.086573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575776.086577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575776.086578:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575776.086582:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3992 00000100:00080000:2.0:1788575776.086587:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575776.086591:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575776.086603:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575776.086619:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495751552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575776.086629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575776.086633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575776.086636:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575776.086642:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3992 00000100:00100000:1.0:1788575776.086644:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495751424:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575776.086646:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575776.086655:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575776.086661:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575776.086674:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495751680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575776.086684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575776.086688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575776.086690:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575776.086694:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3992 00000100:00080000:2.0:1788575776.086703:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575776.086706:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575776.086711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575776.086721:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495751808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575776.086740:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575776.086799:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff93a24940a680 x1875453495751424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.086819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495751424/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.086830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.086834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575776.086838:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495751424:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575776.086868:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751552:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575776.086892:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575776.086905:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff93a249409c00 x1875453495751552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.086915:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495751552/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.086923:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.086926:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575776.086929:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751552:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575776.086935:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751680:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575776.086954:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575776.086975:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff93a249408e00 x1875453495751680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.087003:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495751680/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.087011:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.087014:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575776.087017:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495751680:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575776.087023:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495751808:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575776.087043:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575776.087053:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff93a24940b800 x1875453495751808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.087065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495751808/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.087073:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.087076:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575776.087079:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495751808:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575777.046386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575777.046488:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575778.070350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575778.070405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575779.094345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575779.094419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575780.118343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575780.118412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575781.142367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575781.142432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575781.206413:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.206427:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575781.206432:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.206445:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 3997 00000100:00080000:0.0:1788575781.206458:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.206467:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575781.206517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495751936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.206550:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.206554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575781.206556:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.206562:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3997 00000100:00080000:0.0:1788575781.206566:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.206571:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575781.206583:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575781.206599:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495751936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575781.206600:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495752064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.206630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.206633:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575781.206634:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.206639:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 3997 00000100:00080000:0.0:1788575781.206643:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.206647:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575781.206652:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575781.206662:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495752192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.206671:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.206675:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575781.206677:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575781.206679:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575781.206683:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 3997 00000100:00080000:0.0:1788575781.206687:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.206691:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575781.206696:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575781.206707:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495752320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575781.206735:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff93a2460d0380 x1875453495751936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.206755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495751936/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.206772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.206777:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575781.206781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495751936:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575781.206824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752064:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575781.206846:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575781.206859:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff93a2460d1500 x1875453495752064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.206870:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495752064/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.206878:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.206881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575781.206884:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752064:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575781.206889:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752192:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575781.206916:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575781.206927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff93a2460d2680 x1875453495752192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.206937:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495752192/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.206944:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.206947:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575781.206950:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752192:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575781.206970:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495752320:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575781.206989:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575781.206998:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff93a2460d3480 x1875453495752320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.207008:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495752320/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.207015:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.207018:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575781.207022:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495752320:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575782.166344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575782.166444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575783.190338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575783.190427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575784.214330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575784.214384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575785.238281:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575785.238321:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575786.262325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575786.262391:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575786.326338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575786.326348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575786.326353:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575786.326362:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4002 00000100:00080000:1.0:1788575786.326371:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575786.326378:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575786.326412:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495752448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575786.326443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575786.326447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575786.326448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575786.326452:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4002 00000100:00080000:1.0:1788575786.326456:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575786.326473:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575786.326482:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495752448:192.168.202.102@tcp:250:kworker.0 00000008:00100000:1.0:1788575786.326482:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575786.326497:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495752576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575786.326508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575786.326511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575786.326513:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575786.326517:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4002 00000100:00080000:1.0:1788575786.326521:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575786.326525:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575786.326529:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575786.326536:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495752704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575786.326542:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575786.326545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575786.326548:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575786.326551:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4002 00000100:00080000:1.0:1788575786.326553:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575786.326555:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575786.326556:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575786.326561:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575786.326569:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495752832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575786.326600:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff93a258da1f80 x1875453495752448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.326630:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495752448/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.326640:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.326643:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575786.326647:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495752448:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575786.326677:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752576:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575786.326698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575786.326709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff93a258da0a80 x1875453495752576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.326720:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495752576/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.326728:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.326731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575786.326734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752576:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575786.326740:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752704:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575786.326759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575786.326768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff93a258da2680 x1875453495752704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.326779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495752704/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.326798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.326800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575786.326802:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495752704:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575786.326806:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495752832:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575786.326820:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575786.326826:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff93a258da0e00 x1875453495752832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.326833:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495752832/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.326837:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.326839:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575786.326840:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495752832:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575787.286346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575787.286407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575788.310311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575788.310357:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575789.334331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575789.334373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575790.358354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575790.358457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575791.383204:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575791.385064:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:2.0:1788575791.446354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.446365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575791.446369:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.446379:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4007 00000100:00080000:2.0:1788575791.446398:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.446405:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575791.446447:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495752960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.446487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.446489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.446491:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.446494:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4007 00000100:00080000:2.0:1788575791.446496:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.446499:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575791.446507:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575791.446516:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495753088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.446526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575791.446527:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495752960:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575791.446531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.446534:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.446539:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4007 00000100:00080000:2.0:1788575791.446543:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.446546:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575791.446551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575791.446569:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495753216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.446585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.446589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.446590:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.446594:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4007 00000100:00080000:2.0:1788575791.446606:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.446611:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575791.446617:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575791.446632:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788575791.446633:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495753344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575791.446661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575791.446683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495753344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575792.406750:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575792.407887:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575793.430669:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575793.432018:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575794.455020:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575794.457068:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000400:00000100:1.0:1788575794.518486:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788575794.518532:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575794.518548:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575794.518589:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575794.518598:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788575794.518606:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:0.0:1788575794.518696:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff93a249408700 x1875453495753344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575794.518718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495753344/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575794.518728:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575794.518733:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575794.518737:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495753344:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575794.518743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff93a249409180 x1875453495753088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575794.518769:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495753088/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575794.518776:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575794.518780:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575794.518783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753088:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575794.518789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff93a249408a80 x1875453495753216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575794.518799:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495753216/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575794.518805:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575794.518808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575794.518810:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753216:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575794.518823:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff93a24940ad80 x1875453495752960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575794.518835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495752960/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575794.518843:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575794.518845:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575794.518848:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495752960:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575795.478460:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575795.478663:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575796.502306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575796.502345:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575796.566349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575796.566359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575796.566363:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575796.566373:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4013 00000100:00080000:2.0:1788575796.566383:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575796.566389:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575796.566422:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495753472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575796.566451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575796.566454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575796.566456:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575796.566460:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4013 00000100:00080000:2.0:1788575796.566463:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575796.566467:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575796.566476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575796.566489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495753600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575796.566500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575796.566510:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495753472:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575796.566521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575796.566523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575796.566529:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4013 00000100:00080000:2.0:1788575796.566539:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575796.566544:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575796.566549:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575796.566562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495753728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575796.566572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575796.566577:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1788575796.566579:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575796.566580:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575796.566587:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4013 00000100:00080000:2.0:1788575796.566592:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575796.566596:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575796.566601:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575796.566612:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495753856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575796.566628:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff93a249408000 x1875453495753472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.566646:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495753472/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.566657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.566660:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575796.566664:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495753472:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575796.566725:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753600:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575796.566751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575796.567326:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff93a24940aa00 x1875453495753600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.567362:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495753600/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.567369:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.567372:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575796.567375:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753600:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575796.567382:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753728:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575796.567408:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575796.567423:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff93a249409500 x1875453495753728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.567433:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495753728/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.567440:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.567443:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575796.567445:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495753728:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575796.567450:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495753856:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575796.567468:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575796.567478:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff93a249409880 x1875453495753856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.567489:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495753856/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.567508:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.567511:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575796.567514:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495753856:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575797.526312:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575797.526367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575798.550335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575798.550389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575799.574301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575799.574363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575800.598341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575800.598404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575801.622276:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575801.622367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575801.686337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.686347:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575801.686351:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.686360:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4018 00000100:00080000:0.0:1788575801.686370:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.686376:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575801.686411:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495753984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575801.686437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.686440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575801.686442:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.686445:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4018 00000100:00080000:0.0:1788575801.686448:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.686452:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575801.686461:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575801.686474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495754112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575801.686478:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495753984:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575801.686498:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.686501:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575801.686503:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.686509:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4018 00000100:00080000:0.0:1788575801.686526:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.686531:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788575801.686536:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788575801.686541:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575801.686569:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495754240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575801.686579:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.686583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575801.686584:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.686588:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4018 00000100:00080000:0.0:1788575801.686593:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575801.686593:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff93a2460d2300 x1875453495753984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575801.686600:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575801.686608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575801.686620:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495753984/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575801.686622:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495754368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575801.686642:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.686648:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575801.686653:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495753984:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575801.686676:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754112:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575801.686695:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575801.686705:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff93a2460d1f80 x1875453495754112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.686717:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495754112/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.686725:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.686729:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575801.686732:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754112:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575801.686743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754240:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575801.686760:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575801.686768:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff93a2460d2a00 x1875453495754240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.686779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495754240/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.686787:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.686789:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575801.686792:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754240:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575801.686797:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495754368:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575801.686810:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575801.686818:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff93a2460d1180 x1875453495754368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.686828:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495754368/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.686835:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.686838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575801.686841:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495754368:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575802.646338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575802.646383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575803.670344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575803.670407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575804.694357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575804.694418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575805.718310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575805.718452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575806.742326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575806.742384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575806.806325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575806.806333:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575806.806337:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575806.806352:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4023 00000100:00080000:2.0:1788575806.806362:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575806.806369:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575806.806428:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495754496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575806.806473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575806.806477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575806.806478:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575806.806484:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4023 00000100:00080000:2.0:1788575806.806487:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575806.806491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575806.806500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575806.806505:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495754496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575806.806513:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495754624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575806.806520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575806.806522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575806.806523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575806.806526:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4023 00000100:00080000:2.0:1788575806.806528:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575806.806530:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575806.806533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575806.806540:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495754752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575806.806546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575806.806548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575806.806549:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575806.806552:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4023 00000100:00080000:2.0:1788575806.806559:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575806.806564:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575806.806565:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575806.806571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575806.806579:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495754880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575806.806607:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff93a24940b100 x1875453495754496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575806.806632:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495754496/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575806.806643:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575806.806646:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575806.806650:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495754496:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575806.806672:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754624:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575806.806692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575806.806703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff93a24940bb80 x1875453495754624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575806.806715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495754624/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575806.806723:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575806.806726:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575806.806729:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754624:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575806.806734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754752:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575806.806753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575806.806769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff93a24940b480 x1875453495754752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575806.806780:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495754752/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575806.806798:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575806.806802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575806.806805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495754752:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575806.806816:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495754880:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575806.806835:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575806.806844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff93a249409880 x1875453495754880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575806.806854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495754880/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575806.806862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575806.806864:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575806.806868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495754880:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575807.766334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575807.766409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575808.790342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575808.790525:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575809.814359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575809.814445:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575810.838385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575810.838474:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575811.862309:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575811.862358:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575811.926380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575811.926389:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575811.926392:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575811.926401:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4028 00000100:00080000:0.0:1788575811.926409:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575811.926415:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575811.926447:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495755008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575811.926473:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575811.926477:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575811.926479:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575811.926483:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4028 00000100:00080000:0.0:1788575811.926487:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575811.926491:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575811.926500:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575811.926511:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495755136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575811.926519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575811.926520:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495755008:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575811.926522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575811.926525:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575811.926541:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4028 00000100:00080000:0.0:1788575811.926544:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575811.926547:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575811.926550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575811.926564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495755264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575811.926575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575811.926578:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575811.926580:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575811.926585:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575811.926592:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4028 00000100:00080000:0.0:1788575811.926597:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575811.926602:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575811.926607:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575811.926623:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495755392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575811.926630:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff93a2460d1180 x1875453495755008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575811.926650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495755008/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575811.926657:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575811.926661:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575811.926663:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495755008:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575811.926685:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755136:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575811.926701:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575811.926709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff93a2460d2a00 x1875453495755136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575811.926716:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495755136/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575811.926721:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575811.926723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575811.926725:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755136:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575811.926728:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755264:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575811.926738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575811.926743:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff93a2460d1f80 x1875453495755264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575811.926750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495755264/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575811.926757:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575811.926768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575811.926771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755264:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575811.926776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495755392:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575811.926795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575811.926805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff93a2460d2300 x1875453495755392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575811.926817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495755392/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575811.926822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575811.926824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575811.926826:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495755392:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575812.886395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575812.888066:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575813.910326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575813.910388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575814.934328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575814.934429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575815.958333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575815.958402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575816.982315:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575816.982387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575817.046343:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575817.046354:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575817.046358:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575817.046368:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4033 00000100:00080000:0.0:1788575817.046378:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575817.046400:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575817.046429:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495755520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575817.046456:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575817.046459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575817.046460:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575817.046463:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4033 00000100:00080000:0.0:1788575817.046465:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575817.046467:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575817.046475:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575817.046488:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495755648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575817.046502:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575817.046505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575817.046508:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575817.046510:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495755520:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575817.046515:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4033 00000100:00080000:0.0:1788575817.046520:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575817.046524:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575817.046530:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575817.046542:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495755776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575817.046552:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575817.046555:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575817.046563:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575817.046568:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575817.046571:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4033 00000100:00080000:0.0:1788575817.046576:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575817.046579:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575817.046584:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575817.046605:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495755904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575817.046612:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff93a2460d3480 x1875453495755520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575817.046627:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495755520/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575817.046635:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575817.046638:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575817.046641:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495755520:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575817.046646:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755648:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575817.046666:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575817.046674:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff93a2460d2680 x1875453495755648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575817.046681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495755648/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575817.046696:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575817.046699:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575817.046709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755648:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575817.046734:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755776:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575817.046754:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575817.046763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff93a2460d1500 x1875453495755776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575817.046772:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495755776/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575817.046780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575817.046783:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575817.046791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495755776:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575817.046797:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495755904:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575817.046815:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575817.046824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff93a2460d0380 x1875453495755904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575817.046831:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495755904/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575817.046836:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575817.046838:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575817.046844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495755904:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575818.006325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575818.006414:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575819.030354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575819.030419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575820.054365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575820.054432:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575821.078298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575821.078359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575822.102362:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575822.102429:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575822.166352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575822.166365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575822.166370:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575822.166384:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4038 00000100:00080000:1.0:1788575822.166396:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575822.166405:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575822.166449:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495756032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575822.166482:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575822.166486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575822.166488:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575822.166492:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4038 00000100:00080000:1.0:1788575822.166496:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575822.166501:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575822.166511:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575822.166522:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495756032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575822.166536:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495756160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575822.166547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575822.166550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575822.166551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575822.166556:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4038 00000100:00080000:1.0:1788575822.166560:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575822.166580:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575822.166586:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575822.166588:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575822.166602:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495756288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575822.166613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575822.166616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575822.166618:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575822.166621:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4038 00000100:00080000:1.0:1788575822.166626:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575822.166630:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575822.166635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff93a258da0700 x1875453495756032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575822.166635:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575822.166651:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495756416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575822.166655:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495756032/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575822.166680:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575822.166683:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575822.166687:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495756032:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575822.166710:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756160:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575822.166725:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575822.166733:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff93a258da1c00 x1875453495756160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575822.166747:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495756160/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575822.166753:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575822.166756:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575822.166758:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756160:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575822.166763:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756288:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575822.166783:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575822.166795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff93a258da3800 x1875453495756288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575822.166818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495756288/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575822.166823:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575822.166825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575822.166827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756288:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575822.166831:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495756416:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575822.166846:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575822.166852:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff93a258da3480 x1875453495756416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575822.166859:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495756416/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575822.166872:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575822.166881:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575822.166883:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495756416:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575823.126355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575823.126417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575824.150354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575824.150460:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575825.174353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575825.174601:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575826.198316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575826.198369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575827.222339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575827.222393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575827.286352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.286363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575827.286367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.286377:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4043 00000100:00080000:2.0:1788575827.286387:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.286394:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575827.286430:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495756544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575827.286455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.286459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575827.286461:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.286465:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4043 00000100:00080000:2.0:1788575827.286468:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.286472:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575827.286481:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575827.286494:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495756672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575827.286503:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495756544:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575827.286506:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.286529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575827.286531:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.286537:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4043 00000100:00080000:2.0:1788575827.286542:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.286545:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575827.286551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575827.286561:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575827.286564:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495756800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575827.286575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.286579:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575827.286580:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.286584:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4043 00000100:00080000:2.0:1788575827.286587:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.286591:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575827.286595:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575827.286606:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff93a249409500 x1875453495756544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575827.286607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495756928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575827.286640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495756544/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575827.286651:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575827.286655:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575827.286659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495756544:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575827.286679:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756672:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575827.286698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575827.286709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff93a24940aa00 x1875453495756672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575827.286720:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495756672/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575827.286735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575827.286739:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575827.286743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756672:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575827.286749:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756800:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575827.286767:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575827.286777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff93a249408000 x1875453495756800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575827.286788:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495756800/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575827.286795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575827.286798:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575827.286801:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495756800:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575827.286806:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495756928:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575827.286821:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575827.286830:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff93a24940ad80 x1875453495756928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575827.286866:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495756928/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575827.286874:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575827.286876:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575827.286886:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495756928:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575828.246371:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575828.246439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575829.270321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575829.270416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575830.294332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575830.294386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575831.318310:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575831.318369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575832.342325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575832.342390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575832.406327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575832.406338:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575832.406342:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575832.406352:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4048 00000100:00080000:2.0:1788575832.406361:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575832.406367:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575832.406401:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495757056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575832.406446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575832.406450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575832.406452:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575832.406457:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4048 00000100:00080000:2.0:1788575832.406461:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575832.406465:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575832.406475:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575832.406492:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495757056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575832.406503:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495757184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575832.406515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575832.406519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575832.406521:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575832.406525:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4048 00000100:00080000:2.0:1788575832.406531:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575832.406535:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575832.406540:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575832.406552:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495757312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575832.406559:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575832.406568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575832.406573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575832.406575:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575832.406580:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4048 00000100:00080000:2.0:1788575832.406590:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575832.406595:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575832.406602:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575832.406604:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff93a249408a80 x1875453495757056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575832.406617:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495757440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575832.406620:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495757056/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575832.406645:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575832.406649:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575832.406653:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495757056:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575832.406670:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757184:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575832.406684:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575832.406692:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff93a249409180 x1875453495757184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575832.406715:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495757184/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575832.406720:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575832.406723:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575832.406734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757184:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575832.406743:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757312:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575832.406763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575832.406771:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff93a249408700 x1875453495757312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575832.406778:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495757312/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575832.406783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575832.406785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575832.406787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757312:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575832.406790:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495757440:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575832.406801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575832.406806:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff93a24940b800 x1875453495757440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575832.406812:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495757440/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575832.406816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575832.406818:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575832.406819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495757440:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575833.366300:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575833.366370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575834.390345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575834.390412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575835.414332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575835.414383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575836.438343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575836.438398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575837.462323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575837.462394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575837.526351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575837.526363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575837.526367:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575837.526374:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4053 00000100:00080000:2.0:1788575837.526382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575837.526387:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575837.526417:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495757568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575837.526442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575837.526446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575837.526448:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575837.526451:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4053 00000100:00080000:2.0:1788575837.526455:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575837.526459:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575837.526468:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575837.526483:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495757568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575837.526483:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495757696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575837.526498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575837.526521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575837.526523:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575837.526528:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4053 00000100:00080000:2.0:1788575837.526533:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575837.526537:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575837.526542:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575837.526544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575837.526573:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495757824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575837.526581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575837.526583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575837.526585:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575837.526589:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4053 00000100:00080000:2.0:1788575837.526598:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575837.526602:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575837.526607:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575837.526613:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff93a249408e00 x1875453495757568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575837.526622:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495757952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575837.526631:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495757568/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575837.526643:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575837.526647:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575837.526652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495757568:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575837.526675:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757696:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575837.526694:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575837.526703:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff93a249409c00 x1875453495757696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575837.526718:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495757696/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575837.526727:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575837.526730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575837.526733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757696:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575837.526739:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757824:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575837.526768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575837.526777:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff93a24940a680 x1875453495757824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575837.526784:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495757824/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575837.526789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575837.526791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575837.526793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495757824:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575837.526796:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495757952:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575837.526806:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575837.526811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff93a24940b800 x1875453495757952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575837.526818:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495757952/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575837.526822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575837.526823:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575837.526825:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495757952:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788575838.486370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575838.486414:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575839.510338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575839.510449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575840.534440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575840.535684:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:0.0:1788575841.558340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575841.558393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575842.582541:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575842.583582:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575842.646474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575842.646483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575842.646488:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575842.646498:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4059 00000100:00080000:2.0:1788575842.646509:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575842.646531:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575842.646562:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495758080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575842.646595:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575842.646599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575842.646601:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575842.646604:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4059 00000100:00080000:2.0:1788575842.646608:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575842.646612:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575842.646620:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575842.646631:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495758208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575842.646640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575842.646643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575842.646644:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575842.646648:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4059 00000100:00080000:2.0:1788575842.646651:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575842.646654:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575842.646658:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575842.646665:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495758336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575842.646674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575842.646677:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575842.646683:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575842.646687:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4059 00000100:00080000:2.0:1788575842.646690:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575842.646693:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575842.646697:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575842.646706:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495758464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575842.646822:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495758080:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788575842.646874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575842.646904:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff93a249408700 x1875453495758080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.646919:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495758080/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.646929:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.646933:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575842.646937:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495758080:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575842.646953:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758208:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575842.646974:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575842.646984:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff93a249409180 x1875453495758208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.646996:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495758208/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.647004:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.647007:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575842.647010:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758208:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575842.647015:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758336:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575842.647030:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575842.647039:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff93a249408a80 x1875453495758336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.647049:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495758336/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.647057:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.647060:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575842.647063:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758336:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575842.647068:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495758464:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575842.647081:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575842.647089:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff93a24940ad80 x1875453495758464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.647105:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495758464/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.647113:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.647116:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575842.647119:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495758464:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575843.606357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575843.606769:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575844.630348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575844.631469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575845.654320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575845.654373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575846.678518:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575846.678697:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575847.702387:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575847.702492:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575847.766361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575847.766372:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575847.766376:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575847.766386:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4064 00000100:00080000:0.0:1788575847.766396:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575847.766404:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575847.766441:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495758592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575847.766467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575847.766470:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575847.766472:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575847.766476:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4064 00000100:00080000:0.0:1788575847.766480:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575847.766484:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575847.766494:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575847.766507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495758720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575847.766515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575847.766518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575847.766519:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495758592:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575847.766520:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575847.766537:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4064 00000100:00080000:0.0:1788575847.766540:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575847.766544:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575847.766550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575847.766564:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495758848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575847.766575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575847.766581:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1788575847.766581:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575847.766585:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575847.766598:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4064 00000100:00080000:0.0:1788575847.766602:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575847.766606:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575847.766611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575847.766625:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495758976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575847.766625:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff93a2460d0a80 x1875453495758592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.766660:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495758592/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.766669:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.766673:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575847.766676:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495758592:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575847.766693:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758720:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575847.766709:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575847.766717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff93a2460d0000 x1875453495758720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.766725:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495758720/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.766738:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.766741:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575847.766743:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758720:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575847.766746:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758848:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575847.766759:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575847.766765:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff93a2460d3b80 x1875453495758848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.766773:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495758848/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.766780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.766783:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575847.766786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495758848:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575847.766791:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495758976:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575847.766809:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575847.766820:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff93a2460d0700 x1875453495758976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.766831:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495758976/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.766838:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.766840:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575847.766850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495758976:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575848.726356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575848.726417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575849.750341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575849.750491:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575850.774331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575850.774422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575851.798314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575851.798367:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575852.822408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575852.822468:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:3.0:1788575852.886358:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575852.886368:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575852.886372:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575852.886382:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4069 00000100:00080000:3.0:1788575852.886401:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575852.886409:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575852.886443:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e680 x1875453495759104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575852.886471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575852.886474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575852.886476:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575852.886480:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4069 00000100:00080000:3.0:1788575852.886484:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575852.886488:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575852.886513:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575852.886524:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495759104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575852.886543:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495759232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575852.886555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575852.886558:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575852.886560:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575852.886565:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4069 00000100:00080000:3.0:1788575852.886568:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575852.886571:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575852.886575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575852.886583:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495759360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575852.886590:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788575852.886594:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575852.886597:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575852.886598:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575852.886601:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4069 00000100:00080000:3.0:1788575852.886604:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575852.886608:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575852.886612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575852.886622:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495759488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575852.886637:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff93a25064e680 x1875453495759104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575852.886657:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e680 x1875453495759104/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575852.886681:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575852.886686:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575852.886689:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495759104:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575852.886708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759232:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575852.886733:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575852.886745:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff93a25064fb80 x1875453495759232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575852.886757:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495759232/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575852.886765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575852.886768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575852.886772:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759232:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575852.886777:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759360:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575852.886795:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575852.886808:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff93a25064f480 x1875453495759360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575852.886815:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495759360/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575852.886820:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575852.886822:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575852.886824:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759360:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575852.886827:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495759488:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575852.886838:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575852.886844:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff93a25064c000 x1875453495759488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575852.886850:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495759488/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575852.886854:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575852.886855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575852.886857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495759488:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575853.846322:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575853.846389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575854.870382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575854.870436:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575855.894357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575855.894438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575856.918334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575856.918411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575857.942994:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575857.943198:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575858.006336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.006344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575858.006347:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.006355:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4074 00000100:00080000:2.0:1788575858.006369:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.006378:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575858.006421:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495759616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.006447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.006451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575858.006453:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.006457:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4074 00000100:00080000:2.0:1788575858.006462:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.006467:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575858.006476:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575858.006489:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495759744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.006501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575858.006502:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495759616:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575858.006508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575858.006512:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.006517:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4074 00000100:00080000:2.0:1788575858.006520:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.006524:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575858.006530:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575858.006543:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495759872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.006554:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.006557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575858.006559:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.006578:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4074 00000100:00080000:2.0:1788575858.006582:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.006585:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575858.006591:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575858.006602:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495760000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575858.006621:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575858.006652:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575858.006671:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495760000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575858.966944:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575858.968495:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575859.991128:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575859.991211:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575861.014521:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575861.015795:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788575861.078502:0:2343:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788575861.078536:0:2343:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575861.078549:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575861.078590:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575861.078599:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575861.078608:0:2343:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575861.078712:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff93a24940aa00 x1875453495759744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575861.078734:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495759744/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575861.078745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575861.078750:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575861.078753:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759744:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575861.078761:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff93a249409500 x1875453495759872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575861.078777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495759872/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575861.078784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575861.078787:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575861.078789:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495759872:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575861.078795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff93a249409880 x1875453495760000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575861.078804:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495760000/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575861.078811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575861.078814:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575861.078817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495760000:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575861.078842:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff93a249408000 x1875453495759616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575861.078856:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495759616/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575861.078864:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575861.078867:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575861.078870:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495759616:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575862.038340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575862.038442:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575863.062337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575863.062389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575863.126456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.126467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575863.126471:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.126481:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4079 00000100:00080000:2.0:1788575863.126490:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.126496:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575863.126528:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495760128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575863.126557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.126561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575863.126562:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.126566:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4079 00000100:00080000:2.0:1788575863.126570:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.126582:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575863.126605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575863.126609:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495760128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575863.126621:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495760256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575863.126634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.126650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575863.126652:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.126658:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4079 00000800:00000100:1.0:1788575863.126662:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575863.126663:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.126670:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575863.126676:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575863.126689:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495760384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575863.126709:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff93a24940b480 x1875453495760128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575863.126716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.126726:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575863.126730:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.126735:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4079 00000100:00100000:1.0:1788575863.126740:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495760128/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575863.126742:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.126751:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00080000:1.0:1788575863.126759:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788575863.126760:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788575863.126768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788575863.126776:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495760512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575863.126785:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495760128:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575863.126813:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760256:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575863.126836:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575863.126846:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff93a24940bb80 x1875453495760256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575863.126854:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495760256/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575863.126865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575863.126869:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575863.126872:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760256:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575863.126878:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760384:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575863.126900:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575863.126927:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff93a24940b100 x1875453495760384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575863.126938:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495760384/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575863.126946:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575863.126949:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575863.126952:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760384:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575863.126958:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495760512:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575863.126987:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575863.126995:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff93a249408000 x1875453495760512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575863.127003:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495760512/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575863.127010:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575863.127013:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575863.127023:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495760512:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575864.086388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575864.086906:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575865.110413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575865.113014:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575866.134367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575866.134535:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575867.158347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575867.158423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575868.182308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575868.182349:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575868.246445:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.246457:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575868.246461:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.246470:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4084 00000100:00080000:1.0:1788575868.246480:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575868.246487:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575868.246521:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495760640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575868.246607:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495760640:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575868.246641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.246646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575868.246648:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.246654:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4084 00000100:00080000:1.0:1788575868.246659:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575868.246664:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:0.0:1788575868.246669:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:1.0:1788575868.246676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575868.246693:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495760768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575868.246704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.246708:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575868.246709:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.246714:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4084 00000100:00080000:1.0:1788575868.246718:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575868.246720:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff93a258da3100 x1875453495760640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575868.246739:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495760640/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575868.246741:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575868.246749:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1788575868.246754:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575868.246760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575868.246764:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495760896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575868.246768:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495760640:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575868.246782:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.246785:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575868.246787:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.246792:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4084 00000100:00080000:1.0:1788575868.246796:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575868.246798:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760768:192.168.202.102@tcp:38:kworker.0 00000100:00080000:1.0:1788575868.246803:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575868.246809:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575868.246822:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495761024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575868.246830:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575868.246848:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff93a258da2a00 x1875453495760768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575868.246890:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495760768/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575868.246899:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575868.246903:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575868.246906:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760768:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575868.246967:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760896:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575868.246988:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575868.247000:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff93a258da1880 x1875453495760896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575868.247026:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495760896/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575868.247034:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575868.247036:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575868.247040:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495760896:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575868.247055:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495761024:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575868.247075:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575868.247087:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff93a258da1180 x1875453495761024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575868.247098:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495761024/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575868.247105:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575868.247108:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575868.247111:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495761024:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575869.206340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575869.206406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575870.230385:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575870.230403:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575871.254583:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575871.254867:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575872.278344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575872.278395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788575873.302350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575873.303038:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:2.0:1788575873.366378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575873.366387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575873.366390:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575873.366398:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4089 00000100:00080000:2.0:1788575873.366406:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575873.366412:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575873.366441:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495761152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575873.366462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575873.366465:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575873.366466:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575873.366468:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4089 00000100:00080000:2.0:1788575873.366471:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575873.366473:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575873.366480:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575873.366487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495761280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575873.366493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575873.366508:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575873.366510:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575873.366516:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4089 00000100:00100000:0.0:1788575873.366518:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495761152:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575873.366519:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575873.366525:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575873.366532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575873.366545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495761408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575873.366556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575873.366559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575873.366561:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575873.366565:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4089 00000100:00080000:2.0:1788575873.366569:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575873.366572:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575873.366577:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575873.366585:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:2.0:1788575873.366589:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495761536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575873.366640:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff93a249409880 x1875453495761152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575873.366658:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495761152/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575873.366679:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575873.366684:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575873.366688:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495761152:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575873.366695:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761280:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575873.366718:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575873.366732:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff93a249409500 x1875453495761280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575873.366744:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495761280/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575873.366751:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575873.366786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575873.366791:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761280:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575873.366813:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761408:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575873.366835:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575873.366846:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff93a24940aa00 x1875453495761408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575873.366857:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495761408/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575873.366865:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575873.366868:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575873.366871:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761408:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575873.366884:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495761536:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575873.366903:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575873.366915:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff93a24940ad80 x1875453495761536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575873.366926:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495761536/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575873.366933:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575873.366937:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575873.366940:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495761536:192.168.202.102@tcp:38:kworker.0 00010000:00010000:2.0:1788575874.326433:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575874.326490:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575875.350327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575875.350380:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575876.374328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575876.374390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575877.398365:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575877.398455:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575878.422334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575878.422469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575878.486398:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.486408:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575878.486413:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.486423:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4094 00000100:00080000:1.0:1788575878.486432:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.486438:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575878.486473:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495761664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.486500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.486504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.486518:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.486522:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4094 00000100:00080000:1.0:1788575878.486526:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.486531:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575878.486541:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575878.486552:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495761664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575878.486556:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495761792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.486568:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.486571:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.486573:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.486576:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4094 00000100:00080000:1.0:1788575878.486579:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.486583:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575878.486599:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575878.486613:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495761920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575878.486623:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575878.486625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.486630:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.486632:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.486637:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4094 00000100:00080000:1.0:1788575878.486641:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.486646:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575878.486662:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575878.486681:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff93a258da2300 x1875453495761664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575878.486683:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495762048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575878.486706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495761664/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.486726:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.486730:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575878.486734:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495761664:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575878.486754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761792:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575878.486772:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575878.486783:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff93a258da0000 x1875453495761792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.486801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495761792/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.486808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.486811:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575878.486814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761792:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575878.486819:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761920:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575878.486836:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575878.486845:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff93a258da0380 x1875453495761920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.486855:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495761920/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.486862:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.486865:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575878.486868:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495761920:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575878.486873:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495762048:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575878.486890:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575878.486899:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff93a258da1180 x1875453495762048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.486909:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495762048/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.486928:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.486931:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575878.486934:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495762048:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575879.446352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575879.446413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575880.470325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575880.470385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575881.494352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575881.494412:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575882.518342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575882.518398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575883.542364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575883.542399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575883.606371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575883.606387:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575883.606404:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575883.606412:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4100 00000100:00080000:3.0:1788575883.606421:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575883.606426:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575883.606451:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495762176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575883.606472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575883.606474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575883.606475:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575883.606478:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4100 00000100:00080000:3.0:1788575883.606480:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575883.606482:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575883.606490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575883.606497:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495762304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575883.606503:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575883.606505:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575883.606507:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575883.606510:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4100 00000100:00080000:3.0:1788575883.606511:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575883.606514:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575883.606517:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575883.606526:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495762176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575883.606528:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495762432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575883.606566:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575883.606570:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575883.606572:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575883.606577:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4100 00000100:00080000:3.0:1788575883.606582:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575883.606585:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575883.606589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575883.606601:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495762560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575883.606612:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575883.606654:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff93a25064d500 x1875453495762176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.606671:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495762176/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.606691:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.606695:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575883.606699:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495762176:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575883.606706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762304:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575883.606728:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575883.606740:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff93a25064f100 x1875453495762304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.606750:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495762304/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.606757:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.606760:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575883.606763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762304:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575883.606792:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762432:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575883.606813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575883.606825:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff93a25064ce00 x1875453495762432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.606836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495762432/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.606844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.606848:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575883.606851:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762432:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575883.606865:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495762560:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575883.606884:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575883.606892:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff93a25064d180 x1875453495762560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.606905:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495762560/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.606910:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575883.606912:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575883.606914:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495762560:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575884.566338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575884.566390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575885.590342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575885.590484:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575886.614339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575886.614389:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575887.638318:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575887.638365:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575888.662332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575888.662386:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575888.726349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.726356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575888.726360:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.726369:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4105 00000100:00080000:1.0:1788575888.726378:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.726385:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575888.726420:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495762688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575888.726461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.726465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575888.726467:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.726471:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4105 00000100:00080000:1.0:1788575888.726476:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.726481:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575888.726491:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575888.726496:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495762688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575888.726515:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495762816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575888.726524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.726526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575888.726528:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.726531:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4105 00000100:00080000:1.0:1788575888.726540:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575888.726551:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575888.726559:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575888.726565:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575888.726574:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495762944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575888.726586:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff93a258da1880 x1875453495762688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575888.726586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.726589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575888.726591:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.726596:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4105 00000100:00080000:1.0:1788575888.726601:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.726606:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575888.726609:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575888.726614:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495762688/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575888.726618:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495763072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575888.726632:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575888.726637:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575888.726641:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495762688:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575888.726662:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762816:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575888.726677:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575888.726684:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff93a258da2a00 x1875453495762816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575888.726691:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495762816/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.726697:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575888.726698:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575888.726700:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762816:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575888.726703:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762944:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575888.726712:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575888.726717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff93a258da3100 x1875453495762944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575888.726723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495762944/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.726735:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575888.726737:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575888.726739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495762944:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575888.726742:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495763072:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575888.726762:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575888.726769:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff93a258da3480 x1875453495763072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575888.726775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495763072/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.726780:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575888.726781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575888.726783:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495763072:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575889.686313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575889.686359:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575890.710330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788575890.710417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575891.734369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575891.734593:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575892.758342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575892.758511:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575893.782363:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575893.782523:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575893.846360:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575893.846370:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575893.846375:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575893.846385:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4110 00000100:00080000:0.0:1788575893.846395:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575893.846410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575893.846447:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495763200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575893.846485:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575893.846488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575893.846489:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575893.846493:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4110 00000100:00080000:0.0:1788575893.846496:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575893.846500:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575893.846508:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575893.846520:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495763328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575893.846522:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495763200:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575893.846530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575893.846533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575893.846535:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575893.846537:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4110 00000100:00080000:0.0:1788575893.846539:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575893.846541:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575893.846543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575893.846550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495763456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575893.846556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575893.846559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575893.846560:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575893.846562:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4110 00000100:00080000:0.0:1788575893.846580:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575893.846583:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575893.846587:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575893.846592:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575893.846598:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495763584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575893.846636:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff93a2460d0700 x1875453495763200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.846654:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495763200/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.846664:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.846668:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575893.846672:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495763200:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575893.846711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763328:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575893.846729:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575893.846737:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff93a2460d3b80 x1875453495763328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.846745:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495763328/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.846750:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.846752:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575893.846754:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763328:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575893.846757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763456:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575893.846768:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575893.846773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff93a2460d0000 x1875453495763456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.846779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495763456/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.846784:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.846785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575893.846787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763456:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575893.846789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495763584:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575893.846799:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575893.846804:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff93a2460d0a80 x1875453495763584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.846817:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495763584/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.846822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.846824:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575893.846827:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495763584:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575894.806357:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575894.806418:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575895.830347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575895.830405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575896.854359:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575896.854423:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575897.878346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575897.878413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575898.902340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575898.902400:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575898.966336:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.966345:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575898.966349:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.966358:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4115 00000100:00080000:0.0:1788575898.966368:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.966374:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575898.966407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495763712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575898.966432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.966436:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575898.966438:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.966443:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4115 00000100:00080000:0.0:1788575898.966446:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.966449:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575898.966457:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575898.966481:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495763712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575898.966485:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495763840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575898.966497:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.966500:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575898.966502:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.966506:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4115 00000100:00080000:0.0:1788575898.966510:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.966514:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575898.966519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575898.966529:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495763968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575898.966539:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575898.966540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.966544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575898.966546:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.966551:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4115 00000100:00080000:0.0:1788575898.966554:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.966558:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575898.966562:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575898.966573:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495764096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575898.966580:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff93a2460d0380 x1875453495763712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.966599:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495763712/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.966610:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.966613:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575898.966618:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495763712:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575898.966636:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763840:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575898.966656:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575898.966667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff93a2460d1500 x1875453495763840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.966679:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495763840/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.966699:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.966702:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575898.966706:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763840:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575898.966711:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763968:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575898.966731:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575898.966741:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff93a2460d2680 x1875453495763968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.966752:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495763968/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.966773:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.966776:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575898.966779:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495763968:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575898.966786:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495764096:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575898.966803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575898.966813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff93a2460d3480 x1875453495764096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.966824:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495764096/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.966831:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.966834:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575898.966844:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495764096:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575899.926396:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575899.926490:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575900.950323:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575900.950398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575901.974345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575901.974437:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575902.998358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575903.001867:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575904.022355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575904.022443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575904.086373:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575904.086388:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575904.086394:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575904.086407:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4120 00000100:00080000:2.0:1788575904.086421:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575904.086431:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575904.086479:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495764224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575904.086517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575904.086522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575904.086523:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575904.086528:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4120 00000100:00080000:2.0:1788575904.086532:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575904.086536:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575904.086548:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575904.086560:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495764352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575904.086649:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495764224:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575904.086729:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575904.086733:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575904.086735:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575904.086739:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4120 00000100:00080000:2.0:1788575904.086743:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575904.086748:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575904.086753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575904.086766:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495764480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575904.086775:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575904.086779:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575904.086781:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575904.086827:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575904.086878:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff93a249408a80 x1875453495764224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575904.086900:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495764224/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575904.086911:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575904.086915:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575904.086919:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495764224:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575904.086951:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764352:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575904.086975:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575904.086987:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff93a249409180 x1875453495764352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575904.086999:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495764352/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575904.087007:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575904.087010:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575904.087013:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764352:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575904.087019:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764480:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575904.087039:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575904.087059:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff93a249408700 x1875453495764480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575904.087071:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495764480/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575904.087078:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575904.087081:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575904.087085:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764480:192.168.202.102@tcp:8:kworker.0 00000100:00080000:2.0:1788575904.087149:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4120 00000100:00080000:2.0:1788575904.087154:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575904.087158:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575904.087164:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575904.087176:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495764608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575904.087272:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495764608:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575904.087343:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575904.087373:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff93a24940b800 x1875453495764608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575904.087391:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495764608/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575904.087401:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575904.087405:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575904.087409:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495764608:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788575905.046557:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575905.047855:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575906.070337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575906.070488:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575907.094604:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575907.094942:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575908.118431:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575908.118446:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575909.142606:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575909.145322:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575909.207574:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575909.207590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575909.207596:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575909.207607:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4125 00000100:00080000:2.0:1788575909.207644:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575909.207654:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575909.207697:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495764736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575909.207734:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575909.207738:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575909.207740:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575909.207745:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4125 00000100:00080000:2.0:1788575909.207749:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575909.207753:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575909.207765:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575909.207777:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495764864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575909.207787:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575909.207791:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575909.207793:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575909.207797:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4125 00000100:00080000:2.0:1788575909.207800:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575909.207804:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575909.207809:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575909.207819:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495764992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575909.207828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575909.207837:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575909.207838:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575909.207842:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4125 00000100:00080000:2.0:1788575909.207846:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575909.207850:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575909.207856:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575909.207867:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495765120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575909.208100:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495764736:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575909.208177:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575909.208225:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff93a24940a680 x1875453495764736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575909.208299:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495764736/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575909.208310:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575909.208314:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575909.208319:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495764736:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575909.208337:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764864:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575909.208359:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575909.208371:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff93a249409c00 x1875453495764864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575909.208383:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495764864/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575909.208391:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575909.208394:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575909.208398:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764864:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575909.208403:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764992:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575909.208420:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575909.208429:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff93a249408e00 x1875453495764992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575909.208440:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495764992/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575909.208447:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575909.208450:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575909.208453:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495764992:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575909.208459:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495765120:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575909.208479:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575909.208489:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff93a24940b800 x1875453495765120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575909.208508:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495765120/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575909.208517:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575909.208520:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575909.208523:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495765120:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575910.166520:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575910.167252:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575911.190358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575911.190903:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575912.214452:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575912.215385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575913.238370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575913.238457:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575914.262367:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575914.263350:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575914.327263:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.327274:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575914.327278:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.327288:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4130 00000100:00080000:2.0:1788575914.327299:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.327306:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575914.327342:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495765248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.327358:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.327360:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575914.327362:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.327364:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4130 00000100:00080000:2.0:1788575914.327367:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.327371:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575914.327380:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575914.327391:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495765376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.327416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.327419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575914.327420:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.327424:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4130 00000100:00080000:2.0:1788575914.327428:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.327432:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575914.327437:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575914.327449:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495765504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.327458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.327462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575914.327463:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.327467:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4130 00000100:00080000:2.0:1788575914.327470:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.327473:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575914.327477:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575914.327487:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495765632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575914.327589:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495765248:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575914.327647:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575914.327680:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff93a249408700 x1875453495765248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575914.327692:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495765248/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575914.327698:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575914.327701:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575914.327704:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495765248:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575914.327708:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495765376:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575914.327721:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575914.327728:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff93a249409180 x1875453495765376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575914.327742:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495765376/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575914.327747:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575914.327749:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575914.327751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495765376:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575914.327754:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495765504:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575914.327767:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575914.327773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff93a249408a80 x1875453495765504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575914.327779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495765504/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575914.327783:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575914.327785:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575914.327786:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495765504:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575914.327789:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495765632:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575914.327845:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575914.327856:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff93a24940ad80 x1875453495765632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575914.327863:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495765632/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575914.327876:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575914.327878:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575914.327881:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495765632:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575915.286382:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575915.287118:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575916.310718:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575916.310728:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575917.334320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575917.334387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575918.358350:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575918.358398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575919.382311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575919.382372:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575919.446363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575919.446374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575919.446378:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575919.446388:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4135 00000100:00080000:3.0:1788575919.446397:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575919.446405:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788575919.446443:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495765760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575919.446472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575919.446476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575919.446477:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575919.446481:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4135 00000100:00080000:3.0:1788575919.446502:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575919.446506:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575919.446516:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575919.446520:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495765760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575919.446533:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495765888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575919.446545:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575919.446549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575919.446551:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575919.446556:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4135 00000100:00080000:3.0:1788575919.446560:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575919.446563:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575919.446569:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575919.446582:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:3.0:1788575919.446582:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495766016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575919.446608:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575919.446612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575919.446614:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575919.446617:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4135 00000100:00080000:3.0:1788575919.446622:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575919.446625:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788575919.446629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575919.446635:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff93a25064c700 x1875453495765760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575919.446642:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495766144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575919.446665:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495765760/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575919.446692:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575919.446697:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575919.446701:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495765760:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575919.446721:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495765888:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575919.446737:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575919.446746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff93a25064ca80 x1875453495765888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575919.446755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495765888/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575919.446768:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575919.446772:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575919.446775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495765888:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575919.446781:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766016:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575919.446801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575919.446812:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff93a25064c380 x1875453495766016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575919.446823:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495766016/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575919.446830:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575919.446833:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575919.446836:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766016:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575919.446842:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495766144:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575919.446860:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575919.446870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff93a25064e300 x1875453495766144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575919.446877:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495766144/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575919.446882:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575919.446884:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575919.446890:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495766144:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575920.406329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575920.406384:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575921.430306:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575921.430330:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575922.454327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575922.454402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575923.478343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575923.478441:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575924.502366:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575924.502466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575924.566371:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.566380:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575924.566384:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.566395:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4141 00000100:00080000:0.0:1788575924.566404:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.566411:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575924.566446:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495766272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575924.566480:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.566483:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575924.566485:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.566489:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4141 00000100:00080000:0.0:1788575924.566492:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.566495:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575924.566504:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575924.566519:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495766400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575924.566528:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495766272:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575924.566531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.566551:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575924.566553:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.566558:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4141 00000100:00080000:0.0:1788575924.566563:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.566567:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575924.566572:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575924.566583:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495766528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575924.566592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.566595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575924.566597:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.566601:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4141 00000100:00080000:0.0:1788575924.566605:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.566608:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575924.566612:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575924.566622:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495766656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575924.566631:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575924.566661:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575924.566688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495766656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575925.526483:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575925.528803:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575926.550943:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575926.551278:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575927.574525:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575927.575757:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788575927.639099:0:2341:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:2.0:1788575927.639144:0:2341:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575927.639161:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575927.639208:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575927.639292:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:2.0:1788575927.639301:0:2341:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575927.639383:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff93a2460d1180 x1875453495766656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575927.639406:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495766656/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575927.639418:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575927.639422:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575927.639426:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495766656:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575927.639434:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff93a2460d1f80 x1875453495766400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575927.639461:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495766400/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575927.639469:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575927.639472:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575927.639476:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766400:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575927.639482:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff93a2460d2a00 x1875453495766528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575927.639493:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495766528/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575927.639500:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575927.639503:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575927.639505:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766528:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575927.639511:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff93a2460d2300 x1875453495766272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575927.639522:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495766272/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575927.639529:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575927.639532:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575927.639534:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495766272:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575928.598398:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575928.598559:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575929.622328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575929.622381:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575929.686346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575929.686369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575929.686373:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575929.686383:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4146 00000100:00080000:1.0:1788575929.686393:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575929.686400:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575929.686429:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495766784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575929.686456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575929.686460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575929.686463:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575929.686467:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4146 00000100:00080000:1.0:1788575929.686471:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575929.686475:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575929.686485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575929.686490:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495766784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575929.686501:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495766912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575929.686512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575929.686516:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575929.686518:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575929.686528:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4146 00000100:00080000:1.0:1788575929.686542:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575929.686545:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575929.686551:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575929.686553:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575929.686564:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495767040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575929.686582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575929.686586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575929.686588:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575929.686592:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4146 00000100:00080000:1.0:1788575929.686597:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575929.686601:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575929.686606:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff93a258da3800 x1875453495766784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575929.686608:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575929.686625:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495767168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575929.686631:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495766784/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575929.686645:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575929.686650:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575929.686655:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495766784:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575929.686682:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766912:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575929.686705:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575929.686726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff93a258da1c00 x1875453495766912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575929.686737:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495766912/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575929.686744:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575929.686747:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575929.686751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495766912:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575929.686757:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767040:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575929.686778:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575929.686789:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff93a258da0700 x1875453495767040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575929.686798:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495767040/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575929.686816:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575929.686849:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575929.686852:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767040:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575929.686858:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495767168:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575929.686882:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575929.686894:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff93a258da0e00 x1875453495767168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575929.686904:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495767168/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575929.686911:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575929.686914:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575929.686917:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495767168:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575930.646335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575930.646385:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575931.670319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575931.670361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575932.694334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575932.694383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575933.718341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575933.718379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575934.742339:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575934.742409:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575934.806438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.806448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575934.806451:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.806461:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4151 00000100:00080000:2.0:1788575934.806471:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.806478:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575934.806514:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495767296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575934.806588:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495767296:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788575934.806653:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575934.806838:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.806843:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575934.806845:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.806851:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4151 00000100:00080000:2.0:1788575934.806855:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.806860:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575934.806866:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff93a24940aa00 x1875453495767296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575934.806870:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575934.806889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495767296/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575934.806891:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495767424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575934.806904:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.806907:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575934.806909:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.806913:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4151 00000100:00080000:2.0:1788575934.806917:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.806931:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575934.806937:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575934.806949:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495767552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575934.806959:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.806962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575934.806964:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.806967:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4151 00000100:00080000:2.0:1788575934.806971:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.806974:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575934.806979:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575934.806988:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495767680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575934.807013:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.807018:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575934.807023:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495767296:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575934.807045:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767424:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575934.807066:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575934.807078:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff93a249409500 x1875453495767424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.807089:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495767424/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.807108:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.807111:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575934.807114:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767424:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575934.807119:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767552:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575934.807136:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575934.807144:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff93a249409880 x1875453495767552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.807154:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495767552/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.807162:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.807164:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575934.807167:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767552:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575934.807177:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495767680:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575934.807194:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575934.807203:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff93a249408000 x1875453495767680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.807213:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495767680/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.807220:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.807223:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575934.807226:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495767680:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575935.766340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575935.766410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575936.790424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575936.791178:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575937.814326:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575937.814382:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575938.838332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575938.838377:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575939.862327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575939.862396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575939.926344:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575939.926352:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575939.926356:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575939.926364:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4156 00000100:00080000:0.0:1788575939.926373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575939.926379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575939.926410:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495767808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575939.926432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575939.926448:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575939.926449:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575939.926452:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4156 00000100:00080000:0.0:1788575939.926455:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575939.926458:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575939.926466:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575939.926474:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495767936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575939.926486:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495767808:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575939.926493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575939.926497:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575939.926499:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575939.926505:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4156 00000100:00080000:0.0:1788575939.926509:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575939.926513:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575939.926519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575939.926531:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495768064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575939.926537:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575939.926540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575939.926543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575939.926545:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575939.926549:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4156 00000100:00080000:0.0:1788575939.926554:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575939.926556:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575939.926565:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575939.926572:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575939] req@ffff93a2460d2300 x1875453495767808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575939.926575:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495768192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575939.926599:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495767808/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575939.926607:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575939.926610:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575939.926613:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495767808:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575939.926634:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767936:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575939.926654:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575939.926667:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575939] req@ffff93a2460d2a00 x1875453495767936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575939.926682:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495767936/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575939.926688:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575939.926690:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575939.926691:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495767936:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575939.926695:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768064:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575939.926710:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575939.926716:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575939] req@ffff93a2460d1f80 x1875453495768064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575939.926723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495768064/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575939.926742:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575939.926744:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575939.926746:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768064:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575939.926750:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495768192:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575939.926763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575939.926770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575939] req@ffff93a2460d1180 x1875453495768192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575939.926777:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495768192/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575939.926789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575939.926791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575939.926793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495768192:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575940.886301:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575940.886353:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575941.910342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575941.910395:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575942.934308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575942.934338:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575943.958333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575943.958387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575944.982332:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575944.982390:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575945.046300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575945.046307:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575945.046314:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575945.046322:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4161 00000100:00080000:0.0:1788575945.046330:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575945.046336:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575945.046367:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495768320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575945.046392:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575945.046395:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575945.046396:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575945.046399:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4161 00000100:00080000:0.0:1788575945.046401:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575945.046404:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575945.046411:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575945.046419:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495768448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575945.046442:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495768320:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575945.046443:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575945.046449:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575945.046452:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575945.046456:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4161 00000100:00080000:0.0:1788575945.046459:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575945.046461:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575945.046465:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575945.046473:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495768576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575945.046479:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575945.046482:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575945.046483:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575945.046485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4161 00000100:00080000:0.0:1788575945.046494:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575945.046496:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575945.046499:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575945.046503:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575945.046507:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495768704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575945.046537:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff93a2460d3480 x1875453495768320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575945.046550:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495768320/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575945.046557:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575945.046559:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575945.046562:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495768320:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575945.046578:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768448:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575945.046591:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575945.046598:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff93a2460d2680 x1875453495768448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575945.046604:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495768448/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575945.046609:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575945.046610:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575945.046612:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768448:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575945.046623:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768576:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575945.046636:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575945.046642:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff93a2460d1500 x1875453495768576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575945.046648:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495768576/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575945.046653:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575945.046654:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575945.046656:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768576:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575945.046659:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495768704:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575945.046668:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575945.046673:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff93a2460d0380 x1875453495768704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575945.046705:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495768704/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575945.046710:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575945.046712:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575945.046714:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495768704:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575946.006343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575946.006393:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575947.030424:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575947.032767:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575948.054335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575948.054431:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575949.078328:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575949.078422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575950.102343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575950.102440:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575950.166353:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.166363:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575950.166367:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.166377:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4166 00000100:00080000:2.0:1788575950.166386:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.166392:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575950.166433:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b100 x1875453495768832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575950.166458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.166461:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575950.166476:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.166480:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4166 00000100:00080000:2.0:1788575950.166485:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.166489:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575950.166498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575950.166503:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495768832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:2.0:1788575950.166517:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940bb80 x1875453495768960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575950.166529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.166532:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575950.166534:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.166538:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4166 00000100:00080000:2.0:1788575950.166543:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.166546:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575950.166551:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575950.166562:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b480 x1875453495769088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575950.166566:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575950.166584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.166588:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575950.166589:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.166593:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4166 00000100:00080000:2.0:1788575950.166597:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.166601:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575950.166606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575950.166609:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff93a24940b100 x1875453495768832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575950.166626:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495769216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575950.166651:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b100 x1875453495768832/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575950.166660:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575950.166663:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575950.166666:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495768832:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575950.166687:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768960:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575950.166711:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575950.166723:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff93a24940bb80 x1875453495768960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575950.166735:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940bb80 x1875453495768960/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575950.166741:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575950.166743:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575950.166745:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495768960:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575950.166748:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769088:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575950.166763:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575950.166770:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff93a24940b480 x1875453495769088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575950.166776:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b480 x1875453495769088/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575950.166789:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575950.166791:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575950.166793:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769088:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575950.166804:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495769216:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575950.166822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575950.166828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff93a249408000 x1875453495769216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575950.166835:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495769216/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575950.166839:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575950.166841:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575950.166843:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495769216:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575951.126378:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575951.126443:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575952.150334:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575952.150405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575953.174341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575953.174396:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575954.198383:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575954.198495:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575955.222386:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575955.222754:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575955.286482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575955.286496:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575955.286501:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575955.286514:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4171 00000100:00080000:0.0:1788575955.286525:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575955.286532:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575955.286574:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495769344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575955.286636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575955.286640:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575955.286642:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575955.286647:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4171 00000100:00080000:0.0:1788575955.286651:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575955.286655:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575955.286666:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575955.286681:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495769472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575955.286692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575955.286695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575955.286698:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575955.286701:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495769344:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575955.286722:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4171 00000100:00080000:0.0:1788575955.286728:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575955.286733:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575955.286739:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575955.286752:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495769600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575955.286763:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575955.286766:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575955.286769:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575955.286781:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4171 00000100:00080000:0.0:1788575955.286786:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575955.286789:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575955.286794:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575955.286800:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575955.286813:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495769728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575955.286857:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff93a2460d0a80 x1875453495769344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575955.286879:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495769344/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575955.286891:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575955.286895:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575955.286899:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495769344:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575955.286937:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769472:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575955.286962:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575955.286973:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff93a2460d0000 x1875453495769472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575955.286984:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495769472/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575955.286992:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575955.286995:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575955.286998:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769472:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575955.287009:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769600:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575955.287030:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575955.287041:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff93a2460d3b80 x1875453495769600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575955.287052:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495769600/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575955.287059:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575955.287063:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575955.287066:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769600:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575955.287072:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495769728:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575955.287091:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575955.287102:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff93a2460d0700 x1875453495769728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575955.287120:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495769728/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575955.287129:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575955.287133:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575955.287136:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495769728:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575956.246295:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575956.246422:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575957.270356:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575957.270613:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575958.294409:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575958.297404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575959.318379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575959.318503:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575960.342447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575960.342453:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788575960.406350:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.406362:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575960.406366:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.406376:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4176 00000100:00080000:0.0:1788575960.406386:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.406410:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575960.406444:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495769856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.406475:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.406479:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.406481:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.406485:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4176 00000100:00080000:0.0:1788575960.406489:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.406493:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575960.406502:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.406514:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495769984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.406522:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.406525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.406527:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.406531:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4176 00000100:00080000:0.0:1788575960.406534:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575960.406592:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495769856:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788575960.406651:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575960.406689:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff93a2460d0700 x1875453495769856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575960.406706:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495769856/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575960.406716:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575960.406720:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575960.406723:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495769856:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575960.406730:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769984:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575960.406754:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575960.406766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff93a2460d3b80 x1875453495769984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575960.406779:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495769984/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575960.406799:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575960.406802:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575960.406806:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495769984:192.168.202.102@tcp:38:kworker.0 00000100:00080000:0.0:1788575960.406886:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575960.406893:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.406906:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495770112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.406925:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.406928:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.406936:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.406940:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4176 00000100:00080000:0.0:1788575960.406945:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.406949:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575960.406954:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.406965:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495770240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575960.407030:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495770112:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575960.407054:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575960.407066:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff93a2460d0000 x1875453495770112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575960.407077:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495770112/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575960.407084:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575960.407087:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575960.407090:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495770112:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575960.407096:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495770240:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575960.407115:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575960.407125:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff93a2460d0a80 x1875453495770240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575960.407137:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495770240/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575960.407145:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575960.407148:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575960.407160:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495770240:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788575961.366484:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575961.368687:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788575962.390348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575962.390408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575963.414417:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575963.414452:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575964.438343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575964.438539:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575965.462343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575965.462402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575965.526353:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.526364:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575965.526369:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.526380:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4181 00000100:00080000:0.0:1788575965.526391:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.526396:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575965.526427:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495770368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575965.526451:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.526454:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575965.526455:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.526458:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4181 00000100:00080000:0.0:1788575965.526460:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.526475:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575965.526483:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575965.526512:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495770496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575965.526515:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495770368:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575965.526530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.526533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575965.526535:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.526539:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4181 00000100:00080000:0.0:1788575965.526544:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.526548:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575965.526553:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575965.526563:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495770624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575965.526571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.526574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575965.526576:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575965.526581:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788575965.526585:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4181 00000100:00080000:0.0:1788575965.526590:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.526594:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575965.526598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575965.526608:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495770752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575965.526631:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff93a2460d0380 x1875453495770368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.526650:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495770368/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.526661:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.526665:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575965.526668:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495770368:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575965.526692:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495770496:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575965.526714:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575965.526726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff93a2460d1500 x1875453495770496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.526738:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495770496/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.526745:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.526748:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575965.526751:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495770496:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575965.526756:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495770624:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575965.526785:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575965.526805:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff93a2460d2680 x1875453495770624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.526816:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495770624/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.526822:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.526825:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575965.526828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495770624:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575965.526833:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495770752:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575965.526851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575965.526860:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff93a2460d3480 x1875453495770752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.526871:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495770752/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.526877:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.526880:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575965.526882:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495770752:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575966.486308:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575966.486361:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575967.510340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575967.510405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575968.534304:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575968.534415:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575969.558340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575969.558407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788575970.582399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575970.583294:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00080000:0.0:1788575970.646773:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.646784:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575970.646788:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.646798:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4187 00000100:00080000:0.0:1788575970.646808:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.646814:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575970.646849:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495770880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575970.646877:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.646881:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575970.646882:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.646886:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4187 00000100:00080000:0.0:1788575970.646890:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.646894:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575970.646904:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575970.646916:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495771008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575970.646927:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495770880:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575970.646928:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.646934:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575970.646936:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.646941:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4187 00000100:00080000:0.0:1788575970.646959:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.646964:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575970.646971:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575970.646980:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575970.646985:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495771136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575970.646996:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.646999:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575970.647001:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.647005:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4187 00000100:00080000:0.0:1788575970.647008:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.647012:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575970.647016:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff93a2460d1180 x1875453495770880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575970.647017:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575970.647038:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495770880/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575970.647039:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495771264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575970.647059:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.647063:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575970.647067:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495770880:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575970.647085:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771008:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575970.647107:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575970.647118:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff93a2460d1f80 x1875453495771008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.647131:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495771008/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.647138:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.647142:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575970.647144:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771008:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575970.647148:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771136:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575970.647163:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575970.647173:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff93a2460d2a00 x1875453495771136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.647184:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495771136/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.647191:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.647194:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575970.647197:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771136:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575970.647211:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495771264:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575970.647261:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575970.647273:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff93a2460d2300 x1875453495771264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.647285:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495771264/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.647292:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.647295:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575970.647298:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495771264:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575971.606333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575971.606387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575972.630636:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575972.630728:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575973.654298:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575973.654394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575974.678504:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575974.678506:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575975.702372:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575975.702447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575975.766424:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.766451:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575975.766457:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.766471:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4192 00000100:00080000:0.0:1788575975.766485:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.766495:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575975.766562:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495771392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575975.766587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.766590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575975.766591:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.766594:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4192 00000100:00080000:0.0:1788575975.766598:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.766602:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575975.766611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575975.766621:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495771520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575975.766627:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495771392:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575975.766630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.766637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575975.766639:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.766644:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4192 00000100:00080000:0.0:1788575975.766647:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.766651:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575975.766656:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575975.766667:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495771648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575975.766677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.766681:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575975.766683:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.766693:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4192 00000100:00080000:0.0:1788575975.766697:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.766699:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575975.766704:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575975.766708:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575975.766719:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495771776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575975.766763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff93a2460d2300 x1875453495771392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.766783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495771392/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.766795:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.766800:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575975.766805:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495771392:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575975.766833:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771520:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575975.766856:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575975.766870:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff93a2460d2a00 x1875453495771520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.766889:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495771520/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.766907:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.766910:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575975.766914:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771520:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575975.766920:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771648:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575975.766940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575975.766950:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff93a2460d1f80 x1875453495771648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.766962:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495771648/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.766977:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.766981:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575975.766984:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495771648:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575975.766990:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495771776:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575975.767008:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575975.767019:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff93a2460d1180 x1875453495771776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.767030:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495771776/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.767049:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.767053:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575975.767056:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495771776:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575976.726329:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575976.726397:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575977.750337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575977.750387:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575978.774347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575978.774402:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575979.798317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575979.798370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575980.822333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575980.822405:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575980.886346:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575980.886354:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575980.886357:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575980.886365:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4197 00000100:00080000:2.0:1788575980.886373:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575980.886379:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575980.886407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495771904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575980.886426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575980.886428:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575980.886429:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575980.886432:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4197 00000100:00080000:2.0:1788575980.886434:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575980.886436:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575980.886442:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575980.886450:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495772032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575980.886457:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575980.886459:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575980.886461:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575980.886465:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4197 00000100:00100000:1.0:1788575980.886466:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495771904:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788575980.886467:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575980.886483:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575980.886487:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575980.886495:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495772160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575980.886501:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575980.886504:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575980.886505:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575980.886508:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4197 00000100:00080000:2.0:1788575980.886510:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575980.886512:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575980.886514:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575980.886520:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495772288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575980.886529:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575980.886617:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff93a249409880 x1875453495771904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575980.886634:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495771904/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575980.886644:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575980.886649:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575980.886652:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495771904:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575980.886679:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772032:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575980.886696:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575980.886704:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff93a249409500 x1875453495772032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575980.886711:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495772032/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575980.886717:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575980.886718:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575980.886720:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772032:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575980.886732:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772160:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575980.886753:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575980.886764:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff93a24940aa00 x1875453495772160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575980.886774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495772160/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575980.886782:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575980.886786:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575980.886788:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772160:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575980.886794:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495772288:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575980.886813:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575980.886824:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff93a24940ad80 x1875453495772288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575980.886836:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495772288/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575980.886844:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575980.886847:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575980.886850:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495772288:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575981.846335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575981.846404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575982.870317:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575982.870369:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575983.894313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575983.894363:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575984.918344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575984.918404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575985.942313:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575985.942373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575986.006331:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575986.006343:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575986.006347:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575986.006357:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4202 00000100:00080000:2.0:1788575986.006367:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575986.006374:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788575986.006407:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495772416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575986.006433:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575986.006436:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575986.006451:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575986.006457:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4202 00000100:00080000:2.0:1788575986.006468:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575986.006474:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575986.006475:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495772416:192.168.202.102@tcp:250:kworker.0 00000008:00100000:2.0:1788575986.006483:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575986.006497:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495772544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575986.006507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575986.006510:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575986.006512:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575986.006523:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4202 00000100:00080000:2.0:1788575986.006527:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575986.006531:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575986.006535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575986.006546:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495772672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575986.006552:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788575986.006560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575986.006564:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575986.006566:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575986.006571:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4202 00000100:00080000:2.0:1788575986.006574:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575986.006577:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788575986.006581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575986.006597:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff93a249408a80 x1875453495772416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575986.006600:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495772800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575986.006631:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495772416/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.006641:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.006645:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575986.006649:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495772416:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788575986.006669:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772544:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575986.006689:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575986.006700:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff93a249409180 x1875453495772544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.006709:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495772544/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.006714:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.006716:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575986.006718:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772544:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575986.006724:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772672:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788575986.006751:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575986.006763:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff93a249408700 x1875453495772672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.006774:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495772672/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.006792:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.006795:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575986.006798:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495772672:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575986.006803:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495772800:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788575986.006823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788575986.006834:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff93a24940b800 x1875453495772800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.006845:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495772800/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.006853:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.006856:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575986.006859:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495772800:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575986.966341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575986.966394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575987.990331:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575987.990388:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575989.014333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575989.014375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575990.038352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575990.038523:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575991.062319:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575991.062373:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575991.126385:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575991.126404:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575991.126408:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575991.126419:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4207 00000100:00080000:0.0:1788575991.126429:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575991.126436:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788575991.126468:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495772928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575991.126509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575991.126512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575991.126514:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575991.126517:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4207 00000100:00080000:0.0:1788575991.126521:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575991.126524:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575991.126533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575991.126545:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495773056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575991.126554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575991.126558:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575991.126559:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495772928:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788575991.126561:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575991.126570:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4207 00000100:00080000:0.0:1788575991.126581:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575991.126587:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575991.126593:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575991.126603:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495773184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575991.126613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575991.126616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575991.126617:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575991.126625:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4207 00000100:00080000:0.0:1788575991.126634:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575991.126638:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788575991.126644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575991.126656:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495773312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575991.126657:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575991.126683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575991.126706:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495773312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575992.086333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575992.086425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575993.110321:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575993.110370:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575994.134341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575994.134394:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788575994.198503:0:2340:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:3.0:1788575994.198557:0:2340:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575994.198571:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575994.198624:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575994.198633:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:3.0:1788575994.198641:0:2340:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:1.0:1788575994.199444:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff93a2460d0380 x1875453495773312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.199472:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495773312/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.199497:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.199503:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575994.199508:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495773312:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575994.199517:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff93a2460d2680 x1875453495773056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.199532:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495773056/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.199539:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.199543:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575994.199547:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773056:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788575994.199554:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff93a2460d1500 x1875453495773184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.199565:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495773184/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.199573:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.199577:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575994.199580:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773184:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788575994.199587:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff93a2460d3480 x1875453495772928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.199598:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495772928/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.199611:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.199615:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788575994.199618:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495772928:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788575995.158337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575995.158408:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575996.182348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575996.182401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575996.246333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.246341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575996.246343:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.246354:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4212 00000100:00080000:1.0:1788575996.246363:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.246370:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788575996.246402:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2680 x1875453495773440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575996.246426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.246429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575996.246431:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.246434:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4212 00000100:00080000:1.0:1788575996.246437:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.246445:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575996.246455:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575996.246468:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0a80 x1875453495773568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575996.246475:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495773440:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788575996.246480:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.246485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575996.246500:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.246506:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4212 00000100:00080000:1.0:1788575996.246511:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.246515:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575996.246521:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575996.246535:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1f80 x1875453495773696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575996.246541:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:1.0:1788575996.246550:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.246554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575996.246556:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.246561:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4212 00000100:00080000:1.0:1788575996.246565:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.246569:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788575996.246575:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575996.246582:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff93a258da2680 x1875453495773440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575996.246597:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0e00 x1875453495773824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575996.246608:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2680 x1875453495773440/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575996.246630:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575996.246635:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575996.246639:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495773440:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788575996.246663:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773568:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575996.246683:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575996.246695:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff93a258da0a80 x1875453495773568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575996.246726:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0a80 x1875453495773568/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575996.246734:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575996.246736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575996.246740:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773568:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788575996.246745:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773696:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788575996.246764:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575996.246773:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff93a258da1f80 x1875453495773696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575996.246783:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1f80 x1875453495773696/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575996.246791:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575996.246794:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575996.246796:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495773696:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788575996.246802:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495773824:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788575996.246818:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788575996.246828:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff93a258da0e00 x1875453495773824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575996.246843:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0e00 x1875453495773824/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575996.246860:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575996.246862:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788575996.246865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495773824:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788575997.206324:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575997.206371:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575998.230352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575998.230447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575999.254341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788575999.254399:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576000.278343:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576000.278453:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576001.302346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576001.302416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576001.366539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576001.366550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576001.366554:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576001.366564:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4217 00000100:00080000:0.0:1788576001.366575:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576001.366582:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788576001.366764:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495773952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576001.366838:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495773952:192.168.202.102@tcp:250:kworker.0 00000800:00000100:1.0:1788576001.366912:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:0.0:1788576001.366944:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576001.366949:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576001.366950:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576001.366954:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4217 00000100:00080000:0.0:1788576001.366968:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576001.366971:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576001.366981:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576001.366995:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495774080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576001.367348:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff93a2460d0a80 x1875453495773952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576001.367367:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495773952/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576001.367377:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576001.367386:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576001.367390:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495773952:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576001.367416:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774080:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576001.367442:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576001.367455:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff93a2460d0000 x1875453495774080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576001.367468:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495774080/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576001.367476:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576001.367479:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576001.367481:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774080:192.168.202.102@tcp:38:kworker.0 00000100:00080000:0.0:1788576001.367569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576001.367572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576001.367574:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576001.367578:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4217 00000100:00080000:0.0:1788576001.367581:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576001.367584:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576001.367588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576001.367598:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495774208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576001.367607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576001.367609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576001.367610:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576001.367612:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4217 00000100:00080000:0.0:1788576001.367614:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576001.367616:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576001.367619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576001.367624:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495774336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576001.367723:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774208:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576001.367746:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576001.367758:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff93a2460d3b80 x1875453495774208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576001.367770:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495774208/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576001.367778:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576001.367781:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576001.367784:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774208:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576001.367790:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495774336:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576001.367829:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576001.367841:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff93a2460d0700 x1875453495774336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576001.367852:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495774336/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576001.367859:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576001.367862:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576001.367865:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495774336:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576002.326353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576002.326419:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576003.350360:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576003.350417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576004.374336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576004.374411:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576005.398335:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576005.398410:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576006.422346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576006.422413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576006.486340:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.486350:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576006.486355:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.486381:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4222 00000100:00080000:0.0:1788576006.486392:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.486399:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788576006.486438:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0700 x1875453495774464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576006.486464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.486468:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576006.486470:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.486474:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4222 00000100:00080000:0.0:1788576006.486477:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.486481:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576006.486490:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576006.486504:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3b80 x1875453495774592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576006.486516:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495774464:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788576006.486517:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.486533:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576006.486535:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.486540:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4222 00000100:00080000:0.0:1788576006.486544:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.486549:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576006.486554:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788576006.486580:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576006.486583:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0000 x1875453495774720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576006.486596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.486600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576006.486602:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.486606:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4222 00000100:00080000:0.0:1788576006.486611:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.486615:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788576006.486620:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff93a2460d0700 x1875453495774464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788576006.486621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576006.486640:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0700 x1875453495774464/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576006.486644:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0a80 x1875453495774848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576006.486647:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576006.486653:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576006.486659:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495774464:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576006.486683:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774592:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576006.486698:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576006.486706:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff93a2460d3b80 x1875453495774592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576006.486723:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3b80 x1875453495774592/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576006.486729:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576006.486731:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576006.486733:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774592:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576006.486736:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774720:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576006.486750:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576006.486756:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff93a2460d0000 x1875453495774720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576006.486765:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0000 x1875453495774720/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576006.486772:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576006.486775:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576006.486777:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495774720:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576006.486783:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495774848:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576006.486801:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576006.486810:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff93a2460d0a80 x1875453495774848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576006.486821:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0a80 x1875453495774848/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576006.486840:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576006.486844:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576006.486857:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495774848:192.168.202.102@tcp:38:kworker.0 00010000:00010000:0.0:1788576007.446502:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576007.446615:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788576008.470501:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576008.473106:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576009.494345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576009.494497:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576010.518303:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576010.518796:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576011.542347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576011.542413:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576011.606413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576011.606422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576011.606426:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576011.606436:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4228 00000100:00080000:2.0:1788576011.606446:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576011.606453:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788576011.606498:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408e00 x1875453495774976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576011.606551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576011.606555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576011.606557:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576011.606561:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4228 00000100:00080000:2.0:1788576011.606565:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576011.606569:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576011.606589:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576011.606606:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409c00 x1875453495775104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576011.606607:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495774976:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788576011.606624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576011.606628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576011.606630:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576011.606635:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4228 00000100:00080000:2.0:1788576011.606638:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576011.606658:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576011.606663:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576011.606675:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940a680 x1875453495775232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788576011.606682:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788576011.606691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576011.606695:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576011.606697:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576011.606702:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4228 00000100:00080000:2.0:1788576011.606707:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576011.606711:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576011.606716:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576011.606728:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940b800 x1875453495775360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576011.606734:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff93a249408e00 x1875453495774976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576011.606755:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408e00 x1875453495774976/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576011.606767:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576011.606771:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576011.606775:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495774976:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576011.606796:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775104:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576011.606822:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576011.606833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff93a249409c00 x1875453495775104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576011.606843:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409c00 x1875453495775104/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576011.606852:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576011.606855:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576011.606858:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775104:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576011.606870:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775232:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576011.606888:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576011.606896:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff93a24940a680 x1875453495775232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576011.606907:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940a680 x1875453495775232/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576011.606915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576011.606918:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576011.606921:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775232:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576011.606926:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495775360:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576011.606940:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576011.606948:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff93a24940b800 x1875453495775360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576011.606958:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940b800 x1875453495775360/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576011.606966:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576011.606968:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576011.606971:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495775360:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576012.566346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576012.566404:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576013.590336:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576013.590523:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788576014.614434:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576014.614496:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576015.638478:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576015.639651:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788576016.662388:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576016.662469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576016.726471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.726482:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576016.726487:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.726500:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4233 00000100:00080000:3.0:1788576016.726524:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.726532:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788576016.726585:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064df80 x1875453495775488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.726620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.726624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576016.726626:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.726631:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4233 00000100:00080000:3.0:1788576016.726635:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.726639:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576016.726650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576016.726666:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064e300 x1875453495775616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.726691:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.726695:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576016.726714:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.726721:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4233 00000100:00100000:1.0:1788576016.726722:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495775488:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788576016.726726:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.726733:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576016.726739:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576016.726752:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c380 x1875453495775744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.726764:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.726768:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576016.726769:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.726773:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4233 00000100:00080000:3.0:1788576016.726777:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.726781:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576016.726786:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576016.726796:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ca80 x1875453495775872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788576016.726803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576016.726939:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff93a25064df80 x1875453495775488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576016.726968:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064df80 x1875453495775488/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576016.726979:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576016.726982:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576016.726985:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495775488:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576016.726991:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775616:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576016.727011:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576016.727022:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff93a25064e300 x1875453495775616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576016.727033:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064e300 x1875453495775616/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576016.727042:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576016.727045:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576016.727048:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775616:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576016.727078:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775744:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576016.727094:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576016.727102:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff93a25064c380 x1875453495775744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576016.727113:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c380 x1875453495775744/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576016.727133:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576016.727137:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576016.727140:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495775744:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576016.727146:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495775872:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576016.727179:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576016.727192:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff93a25064ca80 x1875453495775872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576016.727204:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ca80 x1875453495775872/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576016.727212:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576016.727214:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576016.727217:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495775872:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576017.686354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576017.686416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576018.710400:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576018.710653:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576019.734537:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576019.736844:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576020.758316:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576020.758406:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576021.782379:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576021.782542:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576021.846389:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576021.846403:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576021.846409:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576021.846422:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4238 00000100:00080000:3.0:1788576021.846456:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576021.846465:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788576021.846510:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c700 x1875453495776000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576021.846544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576021.846548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576021.846550:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576021.846554:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4238 00000100:00080000:3.0:1788576021.846558:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576021.846562:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576021.846575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576021.846588:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d180 x1875453495776128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576021.846594:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495776000:192.168.202.102@tcp:250:kworker.0 00000100:00080000:3.0:1788576021.846606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576021.846610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576021.846612:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576021.846618:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4238 00000100:00080000:3.0:1788576021.846622:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576021.846626:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576021.846644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576021.846657:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064ce00 x1875453495776256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576021.846668:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576021.846676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576021.846679:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576021.846683:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4238 00000100:00080000:3.0:1788576021.846688:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788576021.846692:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:3.0:1788576021.846696:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576021.846703:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576021.846715:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f100 x1875453495776384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576021.846747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff93a25064c700 x1875453495776000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576021.846768:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c700 x1875453495776000/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576021.846779:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576021.846784:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576021.846787:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495776000:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576021.846849:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776128:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576021.846874:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576021.846887:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff93a25064d180 x1875453495776128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576021.846898:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d180 x1875453495776128/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576021.846904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576021.846919:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576021.846922:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776128:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576021.846928:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776256:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576021.846948:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576021.846957:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff93a25064ce00 x1875453495776256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576021.846968:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064ce00 x1875453495776256/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576021.846975:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576021.846978:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576021.846981:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776256:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576021.846986:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495776384:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576021.847019:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576021.847030:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff93a25064f100 x1875453495776384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576021.847056:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f100 x1875453495776384/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576021.847066:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576021.847069:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576021.847072:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495776384:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576022.806287:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576022.806417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576023.830353:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576023.831224:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576024.854369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576024.854792:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576025.878342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:0.0:1788576025.878427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576026.902370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576026.902438:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576026.966350:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.966358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576026.966362:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.966371:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4243 00000100:00080000:0.0:1788576026.966381:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.966388:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788576026.966421:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495776512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.966444:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.966447:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576026.966448:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.966451:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4243 00000100:00080000:0.0:1788576026.966453:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.966456:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576026.966464:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576026.966475:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495776640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.966486:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788576026.966491:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495776512:192.168.202.102@tcp:250:kworker.0 00000100:00080000:0.0:1788576026.966492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576026.966496:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.966502:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4243 00000100:00080000:0.0:1788576026.966506:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.966525:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576026.966532:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788576026.966544:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576026.966547:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495776768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.966569:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.966573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576026.966575:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788576026.966578:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff93a2460d3480 x1875453495776512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576026.966582:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4243 00000100:00080000:0.0:1788576026.966591:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788576026.966598:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495776512/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576026.966599:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576026.966608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788576026.966611:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576026.966616:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576026.966619:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495776512:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576026.966624:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495776896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576026.966640:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776640:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576026.966657:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576026.966669:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff93a2460d1500 x1875453495776640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576026.966681:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495776640/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576026.966703:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576026.966707:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576026.966710:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776640:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576026.966716:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776768:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576026.966736:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576026.966747:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff93a2460d2680 x1875453495776768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576026.966758:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495776768/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576026.966765:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576026.966768:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576026.966771:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495776768:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576026.966775:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495776896:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576026.966788:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576026.966795:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff93a2460d0380 x1875453495776896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576026.966801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495776896/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576026.966806:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576026.966808:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576026.966814:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495776896:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576027.926333:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576027.926376:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576028.950340:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576028.950383:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576029.974349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576029.974503:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576030.998337:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576030.998398:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576032.022347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576032.023163:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576032.086562:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.086573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576032.086578:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.086590:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4248 00000100:00080000:1.0:1788576032.086599:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.086604:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788576032.086634:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0700 x1875453495777024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.086661:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.086663:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576032.086664:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.086667:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4248 00000100:00080000:1.0:1788576032.086670:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.086672:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576032.086679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576032.086686:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1c00 x1875453495777152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.086692:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.086694:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576032.086695:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.086697:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4248 00000100:00080000:1.0:1788576032.086699:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.086712:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576032.086715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576032.086721:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3800 x1875453495777280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.086726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.086729:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576032.086730:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.086732:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4248 00000100:00080000:1.0:1788576032.086733:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.086735:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576032.086737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576032.086741:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3480 x1875453495777408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576032.086805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495777024:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788576032.086852:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576032.086897:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff93a258da0700 x1875453495777024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.086908:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0700 x1875453495777024/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.086915:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.086917:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576032.086920:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495777024:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576032.086932:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777152:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576032.086948:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576032.086955:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff93a258da1c00 x1875453495777152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.086962:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1c00 x1875453495777152/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.086967:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.086969:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576032.086971:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777152:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788576032.086976:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777280:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788576032.086991:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576032.087001:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff93a258da3800 x1875453495777280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.087012:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3800 x1875453495777280/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.087019:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.087022:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576032.087025:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777280:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788576032.087031:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495777408:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576032.087045:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576032.087051:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff93a258da3480 x1875453495777408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.087065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3480 x1875453495777408/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.087070:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.087071:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576032.087073:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495777408:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576033.046342:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576033.047213:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576034.070405:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576034.070620:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576035.094665:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576035.094707:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576036.118374:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576036.118449:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576037.142392:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576037.142468:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576037.206596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576037.206612:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576037.206617:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576037.206630:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4253 00000100:00080000:0.0:1788576037.206644:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576037.206654:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788576037.206706:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495777536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576037.206739:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576037.206743:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576037.206746:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576037.206752:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4253 00000100:00080000:0.0:1788576037.206756:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576037.206761:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576037.206774:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576037.206776:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495777536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576037.206791:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495777664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576037.206828:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576037.206831:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576037.206833:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576037.206838:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4253 00000100:00080000:0.0:1788576037.206844:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576037.206850:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000800:00000100:1.0:1788576037.206851:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000008:00100000:0.0:1788576037.206866:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576037.206881:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495777792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576037.206893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576037.206897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1788576037.206901:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff93a2460d1180 x1875453495777536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.206902:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576037.206917:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4253 00000100:00080000:0.0:1788576037.206923:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788576037.206925:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495777536/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.206930:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576037.206940:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788576037.206945:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576037.206950:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576037.206956:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495777920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576037.206957:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495777536:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576037.206999:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777664:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576037.207028:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576037.207039:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff93a2460d1f80 x1875453495777664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576037.207051:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495777664/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576037.207062:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576037.207066:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576037.207069:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777664:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576037.207074:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777792:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576037.207090:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576037.207098:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff93a2460d2a00 x1875453495777792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576037.207108:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495777792/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576037.207116:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576037.207118:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576037.207121:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495777792:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576037.207126:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495777920:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576037.207144:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576037.207153:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff93a2460d2300 x1875453495777920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576037.207170:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495777920/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576037.207179:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576037.207181:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576037.207184:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495777920:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576038.166358:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576038.166466:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576039.190369:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576039.190539:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576040.214352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576040.214454:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576041.238448:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576041.238892:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576042.262395:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576042.262477:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576042.326425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576042.326437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576042.326443:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576042.326454:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4258 00000100:00080000:0.0:1788576042.326485:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576042.326492:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788576042.326539:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2300 x1875453495778048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576042.326572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576042.326575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576042.326576:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576042.326580:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4258 00000100:00080000:0.0:1788576042.326583:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576042.326586:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576042.326597:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576042.326607:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2a00 x1875453495778176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576042.326614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576042.326616:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576042.326618:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576042.326621:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4258 00000100:00080000:0.0:1788576042.326624:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576042.326626:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576042.326629:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576042.326635:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495778048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576042.326638:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1f80 x1875453495778304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576042.326658:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576042.326665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576042.326666:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576042.326670:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4258 00000100:00080000:0.0:1788576042.326673:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576042.326675:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576042.326679:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576042.326688:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1180 x1875453495778432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788576042.326727:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576042.326776:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff93a2460d2300 x1875453495778048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576042.326797:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2300 x1875453495778048/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576042.326808:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576042.326813:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576042.326817:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495778048:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576042.326824:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778176:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576042.326858:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576042.326871:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff93a2460d2a00 x1875453495778176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576042.326883:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2a00 x1875453495778176/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576042.326904:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576042.326908:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576042.326912:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778176:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576042.326941:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778304:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576042.326966:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576042.326977:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff93a2460d1f80 x1875453495778304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576042.326988:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1f80 x1875453495778304/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576042.326996:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576042.327000:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576042.327003:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778304:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576042.327009:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495778432:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576042.327028:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576042.327040:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff93a2460d1180 x1875453495778432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576042.327052:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1180 x1875453495778432/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576042.327059:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576042.327062:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576042.327075:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495778432:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576043.286330:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576043.286427:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576044.310352:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576044.312148:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576045.334658:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576045.336087:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576046.358325:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576046.358532:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576047.382364:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576047.383144:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576047.446403:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.446418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576047.446424:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.446437:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4263 00000100:00080000:0.0:1788576047.446450:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.446459:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:0.0:1788576047.446506:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d0380 x1875453495778560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.446537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.446541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576047.446543:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.446548:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4263 00000100:00080000:0.0:1788576047.446556:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.446561:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576047.446578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576047.446590:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495778560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576047.446595:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d2680 x1875453495778688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.446627:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.446631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576047.446633:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.446638:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4263 00000100:00080000:0.0:1788576047.446663:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.446668:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576047.446674:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788576047.446675:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576047.446692:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d1500 x1875453495778816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.446703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.446706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576047.446708:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.446711:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4263 00000100:00080000:0.0:1788576047.446714:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.446718:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:0.0:1788576047.446722:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576047.446726:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff93a2460d0380 x1875453495778560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576047.446734:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a2460d3480 x1875453495778944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576047.446746:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d0380 x1875453495778560/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576047.446756:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576047.446759:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576047.446763:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495778560:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576047.446787:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778688:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576047.446803:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576047.446811:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff93a2460d2680 x1875453495778688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576047.446819:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d2680 x1875453495778688/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576047.446824:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576047.446826:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576047.446828:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778688:192.168.202.102@tcp:38:kworker.0 00000100:00100000:1.0:1788576047.446832:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778816:192.168.202.102@tcp:8:kworker.0 00000800:00000100:1.0:1788576047.446848:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576047.446867:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff93a2460d1500 x1875453495778816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576047.446893:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d1500 x1875453495778816/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576047.446901:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576047.446904:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576047.446908:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495778816:192.168.202.102@tcp:8:kworker.0 00000100:00100000:1.0:1788576047.446914:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495778944:192.168.202.102@tcp:38:kworker.0 00000800:00000100:1.0:1788576047.446934:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:1.0:1788576047.446942:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff93a2460d3480 x1875453495778944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576047.446950:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a2460d3480 x1875453495778944/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576047.446957:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576047.446960:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:1.0:1788576047.446963:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a2460d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495778944:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576048.406370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576048.406425:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576049.430314:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576049.430416:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576050.454347:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576050.454499:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576051.478348:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576051.478462:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576052.502327:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576052.502407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576052.566351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576052.566358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576052.566362:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576052.566372:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4269 00000100:00080000:1.0:1788576052.566382:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576052.566414:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788576052.566457:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da3100 x1875453495779072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576052.566478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576052.566481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576052.566482:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576052.566485:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4269 00000100:00080000:1.0:1788576052.566488:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576052.566490:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576052.566497:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576052.566504:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2a00 x1875453495779200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576052.566510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576052.566512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576052.566513:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576052.566515:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4269 00000100:00080000:1.0:1788576052.566516:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576052.566518:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576052.566520:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576052.566525:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1880 x1875453495779328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576052.566531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576052.566533:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576052.566535:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576052.566543:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4269 00000100:00080000:1.0:1788576052.566547:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576052.566550:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576052.566556:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576052.566561:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495779072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:1.0:1788576052.566570:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495779456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788576052.566621:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576052.566660:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff93a258da3100 x1875453495779072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576052.566691:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da3100 x1875453495779072/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576052.566701:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576052.566706:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576052.566709:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495779072:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576052.566717:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779200:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576052.566738:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576052.566750:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff93a258da2a00 x1875453495779200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576052.566762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2a00 x1875453495779200/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576052.566770:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576052.566773:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576052.566776:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779200:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788576052.566782:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779328:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788576052.566804:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576052.566813:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff93a258da1880 x1875453495779328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576052.566820:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1880 x1875453495779328/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576052.566833:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576052.566835:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576052.566837:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779328:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788576052.566872:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495779456:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576052.566888:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576052.566899:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff93a258da1180 x1875453495779456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576052.566910:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495779456/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576052.566918:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576052.566920:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576052.566924:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495779456:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576053.526320:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576053.526417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576054.550338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576054.550379:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576055.574311:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576055.574534:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576056.599872:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576056.599952:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576057.622349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576057.622439:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576057.686322:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576057.686331:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576057.686336:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576057.686344:0:8:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4274 00000100:00080000:1.0:1788576057.686353:0:8:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576057.686359:0:8:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:1.0:1788576057.686391:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0380 x1875453495779584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576057.686430:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576057.686434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576057.686436:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576057.686440:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4274 00000100:00080000:1.0:1788576057.686443:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576057.686446:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576057.686454:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576057.686467:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da0000 x1875453495779712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576057.686474:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495779584:192.168.202.102@tcp:250:kworker.0 00000100:00080000:1.0:1788576057.686496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576057.686513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576057.686515:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576057.686519:0:8:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4274 00000100:00080000:1.0:1788576057.686528:0:8:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576057.686532:0:8:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576057.686537:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576057.686551:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da2300 x1875453495779840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576057.686562:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576057.686566:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576057.686568:0:8:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576057.686572:0:8:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4274 00000100:00080000:1.0:1788576057.686577:0:8:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788576057.686580:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779712:192.168.202.102@tcp:38:kworker.0 00000100:00080000:1.0:1788576057.686584:0:8:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:1.0:1788576057.686590:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576057.686605:0:8:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a258da1180 x1875453495779968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576057.686610:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788576057.686645:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495779968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576058.646521:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576058.647184:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576059.670349:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576059.670421:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576060.694299:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576060.694375:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788576060.759943:0:2342:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.102:988 00000400:00000100:1.0:1788576060.760009:0:2342:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.102@tcp at host 192.168.202.102 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788576060.760042:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788576060.760094:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788576060.760103:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000800:00000100:1.0:1788576060.760111:0:2342:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.102@tcp 00000100:00100000:2.0:1788576060.760280:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff93a258da1180 x1875453495779968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576060.760304:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da1180 x1875453495779968/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576060.760318:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576060.760323:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788576060.760327:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495779968:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788576060.760334:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff93a258da0000 x1875453495779712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576060.760375:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0000 x1875453495779712/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576060.760393:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576060.760397:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788576060.760400:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779712:192.168.202.102@tcp:38:kworker.0 00000100:00100000:2.0:1788576060.760407:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff93a258da2300 x1875453495779840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576060.760428:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da2300 x1875453495779840/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576060.760435:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576060.760438:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788576060.760442:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495779840:192.168.202.102@tcp:8:kworker.0 00000100:00100000:2.0:1788576060.760449:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff93a258da0380 x1875453495779584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576060.760461:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a258da0380 x1875453495779584/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576060.760468:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576060.760472:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:2.0:1788576060.760475:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a258da0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495779584:192.168.202.102@tcp:250:kworker.0 00000100:00100000:3.0:1788576061.718361:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576061.720356:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576062.742370:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576062.742469:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576062.806476:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.806486:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576062.806489:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.806499:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4279 00000100:00080000:2.0:1788576062.806509:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.806516:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788576062.806550:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408700 x1875453495780096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.806577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.806580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576062.806581:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.806584:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4279 00000100:00080000:2.0:1788576062.806587:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.806590:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576062.806598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576062.806606:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409180 x1875453495780224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.806613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.806616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576062.806618:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.806621:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4279 00000100:00080000:2.0:1788576062.806624:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.806628:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576062.806644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576062.806652:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408a80 x1875453495780352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.806658:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.806660:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576062.806661:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.806664:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4279 00000100:00080000:2.0:1788576062.806666:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.806668:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576062.806670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576062.806675:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940ad80 x1875453495780480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576062.807529:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495780096:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788576062.807591:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576062.807630:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff93a249408700 x1875453495780096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576062.807660:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408700 x1875453495780096/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576062.807671:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576062.807675:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576062.807680:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495780096:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576062.807688:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780224:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576062.807707:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576062.807717:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff93a249409180 x1875453495780224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576062.807727:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409180 x1875453495780224/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576062.807733:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576062.807736:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576062.807739:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780224:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788576062.807744:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780352:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788576062.807758:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576062.807766:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff93a249408a80 x1875453495780352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576062.807775:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408a80 x1875453495780352/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576062.807793:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576062.807796:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576062.807799:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780352:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788576062.807805:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495780480:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576062.807823:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576062.807833:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff93a24940ad80 x1875453495780480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576062.807842:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940ad80 x1875453495780480/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576062.807848:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576062.807852:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576062.807855:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495780480:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576063.766511:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576063.766747:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788576064.790447:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576064.790578:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788576065.814341:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576065.814401:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576066.838345:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576066.838476:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576067.862529:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576067.862600:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576067.926915:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576067.926926:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576067.926930:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576067.926940:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4284 00000100:00080000:3.0:1788576067.926950:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576067.926957:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:3.0:1788576067.926991:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064d500 x1875453495780608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576067.927052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576067.927056:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576067.927058:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576067.927062:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4284 00000100:00080000:3.0:1788576067.927066:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576067.927070:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576067.927079:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576067.927092:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064c000 x1875453495780736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576067.927101:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576067.927113:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576067.927115:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576067.927119:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4284 00000100:00080000:3.0:1788576067.927122:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576067.927126:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576067.927130:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576067.927141:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064f480 x1875453495780864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576067.927150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576067.927153:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576067.927155:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576067.927158:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4284 00000100:00080000:3.0:1788576067.927167:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576067.927170:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:3.0:1788576067.927175:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576067.927184:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a25064fb80 x1875453495780992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576067.927577:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495780608:192.168.202.102@tcp:250:kworker.0 00000800:00000100:0.0:1788576067.927744:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576067.927784:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff93a25064d500 x1875453495780608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576067.927801:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064d500 x1875453495780608/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576067.927811:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576067.927815:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576067.927819:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495780608:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576067.927891:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780736:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576067.927915:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576067.927928:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff93a25064c000 x1875453495780736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576067.927941:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064c000 x1875453495780736/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576067.927948:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576067.927951:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576067.927954:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780736:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788576067.927960:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780864:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788576067.927980:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576067.927990:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff93a25064f480 x1875453495780864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576067.928001:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064f480 x1875453495780864/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576067.928008:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576067.928023:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576067.928025:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495780864:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788576067.928971:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495780992:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576067.929001:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576067.929022:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff93a25064fb80 x1875453495780992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576067.929042:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a25064fb80 x1875453495780992/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576067.929050:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576067.929054:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576067.929056:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a25064fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495780992:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576068.886338:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576068.886545:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788576069.910444:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576069.913623:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00000100:00100000:3.0:1788576070.934344:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576070.934585:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576071.958346:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576071.958417:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576072.982354:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576072.982407:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576073.047643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2f28dd17-6644-45dd-bf11-f8af88abec3c->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576073.047655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576073.047659:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576073.047671:0:37:0:(import.c:545:import_select_connection()) MGC192.168.202.102@tcp: connect to NID 192.168.202.102@tcp last attempt 4289 00000100:00080000:2.0:1788576073.047700:0:37:0:(import.c:630:import_select_connection()) MGC192.168.202.102@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576073.047706:0:37:0:(import.c:665:import_select_connection()) MGC192.168.202.102@tcp: import ffff93a258dda800 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000100:00100000:2.0:1788576073.047743:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a24940aa00 x1875453495781120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576073.047772:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576073.047775:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576073.047780:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576073.047784:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4289 00000100:00080000:2.0:1788576073.047789:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576073.047793:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a251f1d800: import ffff93a24916e000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576073.047802:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576073.047815:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409500 x1875453495781248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576073.047816:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495781120:192.168.202.102@tcp:250:kworker.0 00000100:00080000:2.0:1788576073.047824:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) bf89ae46-999b-46d3-9a62-08dfe43909d0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576073.047828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576073.047829:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576073.047832:0:37:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: connect to NID 192.168.202.102@tcp last attempt 4289 00000100:00080000:2.0:1788576073.047835:0:37:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576073.047838:0:37:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff93a251f1d800: import ffff93a24916a000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576073.047842:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576073.047852:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249409880 x1875453495781376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576073.047860:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 96693705-5e29-4404-974e-e904d01a815e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576073.047868:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576073.047870:0:37:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576073.047873:0:37:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: connect to NID 192.168.202.102@tcp last attempt 4289 00000800:00000100:0.0:1788576073.047876:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00080000:2.0:1788576073.047877:0:37:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576073.047879:0:37:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff93a244fd4000: import ffff93a24a13b000 using connection 192.168.202.102@tcp/192.168.202.102@tcp 00000008:00100000:2.0:1788576073.047884:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683396 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576073.047893:0:37:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff93a249408000 x1875453495781504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576073.047916:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff93a24940aa00 x1875453495781120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576073.047943:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a24940aa00 x1875453495781120/t0(0) o250->MGC192.168.202.102@tcp@192.168.202.102@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576073.047955:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a258dda800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576073.047958:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576073.047961:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a24940aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f28dd17-6644-45dd-bf11-f8af88abec3c:2352:1875453495781120:192.168.202.102@tcp:250:kworker.0 00000100:00100000:0.0:1788576073.047979:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495781248:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576073.048005:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576073.048018:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff93a249409500 x1875453495781248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576073.048065:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409500 x1875453495781248/t0(0) o38->lustre-MDT0000-mdc-ffff93a251f1d800@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576073.048086:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576073.048090:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576073.048093:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495781248:192.168.202.102@tcp:38:kworker.0 00000100:00100000:0.0:1788576073.048099:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495781376:192.168.202.102@tcp:8:kworker.0 00000800:00000100:0.0:1788576073.048130:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576073.048140:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff93a249409880 x1875453495781376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576073.048151:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249409880 x1875453495781376/t0(0) o8->lustre-OST0000-osc-ffff93a251f1d800@192.168.202.102@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576073.048158:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24916a000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576073.048160:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576073.048164:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249409880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bf89ae46-999b-46d3-9a62-08dfe43909d0:2352:1875453495781376:192.168.202.102@tcp:8:kworker.0 00000100:00100000:0.0:1788576073.048180:0:2352:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495781504:192.168.202.102@tcp:38:kworker.0 00000800:00000100:0.0:1788576073.048197:0:2352:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.102@tcp 00000100:00100000:0.0:1788576073.048746:0:2352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff93a249408000 x1875453495781504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576073.048762:0:2352:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff93a249408000 x1875453495781504/t0(0) o38->lustre-MDT0000-mdc-ffff93a244fd4000@192.168.202.102@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576073.048769:0:2352:0:(import.c:65:import_set_state_nolock()) ffff93a24a13b000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576073.048772:0:2352:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.102@tcp failed (-110) 00000100:00100000:0.0:1788576073.048781:0:2352:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff93a249408000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:96693705-5e29-4404-974e-e904d01a815e:2352:1875453495781504:192.168.202.102@tcp:38:kworker.0 00000100:00100000:3.0:1788576074.006355:0:20560:0:(client.c:2716:ptlrpc_set_wait()) set ffff93a258062840 going to sleep for 0 seconds 00010000:00010000:2.0:1788576074.006470:0:20561:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 54149 lines, 54149 kept, 0 dropped, 0 bad.